[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:02 dist-test-slave-mk3h krb5kdc[3307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:02 dist-test-slave-mk3h krb5kdc[3307](info): set up 2 sockets
Sep 27 06:59:02 dist-test-slave-mk3h krb5kdc[3307](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 06:59:02.996289 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:03.106489 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34397
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:34397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:03.184124 3320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.184317 3320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.184340 3320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.185478 3320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:03.185518 3320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.185534 3320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.185545 3320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:03.185592 3320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:03.187000 3320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:34397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:03.187245 3320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.187456 3320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:59:03.192921 3320 server_base.cc:1034] running on GCE node
W20260927 06:59:03.192989 3325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.192996 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.193046 3328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.193389 3320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.193693 3320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.194868 3320 hybrid_clock.cc:648] HybridClock initialized: now 1790492343194816 us; error 85 us; skew 500 ppm
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:03.197336 3320 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:03.197796 3320 webserver.cc:460] Webserver started at http://127.3.57.62:42023/ using document root <none> and password file <none>
I20260927 06:59:03.198016 3320 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.198084 3320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.198233 3320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.199297 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "0eb8386c96a34cf4ab6ac5df364fb783"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "8ed3020fafb5b2692d6dc30e47766d52"
server_key_iv: "9aa6f44f0d1d7f7fbfbc2d4813733dab"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.199702 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "0eb8386c96a34cf4ab6ac5df364fb783"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "8ed3020fafb5b2692d6dc30e47766d52"
server_key_iv: "9aa6f44f0d1d7f7fbfbc2d4813733dab"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.201056 3320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:03.201839 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.202047 3320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:03.202152 3320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "0eb8386c96a34cf4ab6ac5df364fb783"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "8ed3020fafb5b2692d6dc30e47766d52"
server_key_iv: "9aa6f44f0d1d7f7fbfbc2d4813733dab"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.202239 3320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.218603 3320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.220413 3320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.220570 3320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.224458 3320 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:34397
I20260927 06:59:03.224509 3387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:34397 every 8 connection(s)
I20260927 06:59:03.224855 3320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:03.225672 3388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:03.227842 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: Bootstrap starting.
I20260927 06:59:03.228389 3388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:03.228660 3388 log.cc:824] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:03.229290 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: No bootstrap required, opened a new log
I20260927 06:59:03.230808 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3320
I20260927 06:59:03.230914 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:03.231081 3388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } }
I20260927 06:59:03.231245 3388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:03.231273 3388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb8386c96a34cf4ab6ac5df364fb783, State: Initialized, Role: FOLLOWER
I20260927 06:59:03.231415 3388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } }
I20260927 06:59:03.231485 3388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:03.231515 3388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:03.231585 3388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:03.231720 3300 external_mini_cluster.cc:1428] Setting key a4f92825859f98430747e9246d5c4778
I20260927 06:59:03.232242 3388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } }
I20260927 06:59:03.232373 3388 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb8386c96a34cf4ab6ac5df364fb783; no voters:
I20260927 06:59:03.232539 3388 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:03.232699 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:03.232776 3388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:03.232939 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [term 1 LEADER]: Becoming Leader. State: Replica: 0eb8386c96a34cf4ab6ac5df364fb783, State: Running, Role: LEADER
I20260927 06:59:03.233085 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } }
I20260927 06:59:03.233616 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb8386c96a34cf4ab6ac5df364fb783. Latest consensus state: current_term: 1 leader_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } } }
I20260927 06:59:03.233700 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:03.234170 3391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb8386c96a34cf4ab6ac5df364fb783" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34397 } } }
I20260927 06:59:03.234370 3391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783 [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:03.234572 3410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:03.234645 3410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:03.234716 3402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:03.234870 3402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:03.236260 3402 catalog_manager.cc:1347] Generated new cluster ID: b1d612686a154adcb126fa4635c1db0c
I20260927 06:59:03.236318 3402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:03.240912 3402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:03.241575 3402 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:03.249181 3402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb8386c96a34cf4ab6ac5df364fb783: Generated new TSK 0
I20260927 06:59:03.249336 3402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:03.249883 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.232580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.3.57.62:34397)
0927 06:59:03.232746 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.232754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.233314 (+ 560us) server_negotiation.cc:407] Connection header received
0927 06:59:03.235912 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.235919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.235970 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.236101 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.238064 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.239393 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.241039 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.241625 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.241741 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.245865 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.245869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.245869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.245879 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.247824 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.248298 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.248301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.248302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.248401 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.248669 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.248670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.248670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.248950 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.249069 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.249482 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.249633 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:03.274230 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:03.351963 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.352166 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.352201 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.353358 3415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.353423 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.353467 3415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:03.354878 3415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:03.355131 3415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.355341 3415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:03.360550 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.360605 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.360847 3415 server_base.cc:1034] running on GCE node
W20260927 06:59:03.360576 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.361121 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.361375 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.362545 3415 hybrid_clock.cc:648] HybridClock initialized: now 1790492343362532 us; error 31 us; skew 500 ppm
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:03.364907 3415 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:03.365321 3415 webserver.cc:460] Webserver started at http://127.3.57.1:40075/ using document root <none> and password file <none>
I20260927 06:59:03.365561 3415 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.365624 3415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.365767 3415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.366757 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "ff0117aab2ea4156924c07a5afda9d41"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "aa975d3e4d4961ec1ee472dca7857ca7"
server_key_iv: "29924f42d945c9035ee54641b0525dfc"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.367092 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "ff0117aab2ea4156924c07a5afda9d41"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "aa975d3e4d4961ec1ee472dca7857ca7"
server_key_iv: "29924f42d945c9035ee54641b0525dfc"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.368438 3415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:03.369135 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.369315 3415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:03.369411 3415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "ff0117aab2ea4156924c07a5afda9d41"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "aa975d3e4d4961ec1ee472dca7857ca7"
server_key_iv: "29924f42d945c9035ee54641b0525dfc"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.369488 3415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.380877 3415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.382560 3415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.382716 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.382931 3415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:03.383231 3415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:03.383284 3415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.383324 3415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:03.383352 3415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.388962 3415 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:33591
I20260927 06:59:03.389010 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:33591 every 8 connection(s)
I20260927 06:59:03.389319 3415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:03.398347 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.390054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:40005 (local address 127.3.57.62:34397)
0927 06:59:03.390280 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.390283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.390421 (+ 138us) server_negotiation.cc:407] Connection header received
0927 06:59:03.390938 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.390939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.390951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.391002 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.392004 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.392507 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.393519 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.393733 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.393808 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.395906 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.395907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.395908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.395915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.397086 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.397484 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.397485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.397485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.397540 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.397802 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.397803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.397803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.397904 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.397956 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.398201 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.398284 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260927 06:59:03.398505 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.389983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34397 (local address 127.3.57.1:40005)
0927 06:59:03.390290 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:03.390332 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:03.390835 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:03.391078 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:03.391082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:03.391719 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:03.391898 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.391902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.392629 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.392630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.393417 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.393419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.393812 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.393813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.393939 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.394394 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:03.394402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:03.395812 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:03.397193 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.397199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.397201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.397404 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.397627 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.397628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.397629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.397735 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.397962 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:03.397966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:03.398091 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 06:59:03.398171 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 06:59:03.398305 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":86,"threads_started":1}
I20260927 06:59:03.398608 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3415
I20260927 06:59:03.398689 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:03.398850 3300 external_mini_cluster.cc:1428] Setting key 80bd771467634bc634ce58f68daf568d
I20260927 06:59:03.399106 3544 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34397
I20260927 06:59:03.399230 3544 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:03.399410 3544 heartbeater.cc:511] Master 127.3.57.62:34397 requested a full tablet report, sending...
I20260927 06:59:03.399966 3352 ts_manager.cc:194] Registered new tserver with Master: ff0117aab2ea4156924c07a5afda9d41 (127.3.57.1:33591)
I20260927 06:59:03.400571 3352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:40005
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:03.422030 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:03.500988 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.501206 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.501230 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.502379 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.502420 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.502446 3549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:03.503870 3549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:03.504117 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.504330 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:03.509116 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.509162 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.509253 3549 server_base.cc:1034] running on GCE node
W20260927 06:59:03.509114 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.509570 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.509817 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.510977 3549 hybrid_clock.cc:648] HybridClock initialized: now 1790492343510957 us; error 31 us; skew 500 ppm
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:03.513312 3549 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:03.513729 3549 webserver.cc:460] Webserver started at http://127.3.57.2:34687/ using document root <none> and password file <none>
I20260927 06:59:03.513965 3549 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.514034 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.514179 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.515141 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "14097b57be0c4d18bbcdee3a8cf84396"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "ee40bf575d924ad33a6aabade57a3c85"
server_key_iv: "e361f3efed9c731534d129f85dfc9eb8"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.515482 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "14097b57be0c4d18bbcdee3a8cf84396"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "ee40bf575d924ad33a6aabade57a3c85"
server_key_iv: "e361f3efed9c731534d129f85dfc9eb8"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.516839 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:03.517596 3564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.517781 3549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:03.517876 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "14097b57be0c4d18bbcdee3a8cf84396"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "ee40bf575d924ad33a6aabade57a3c85"
server_key_iv: "e361f3efed9c731534d129f85dfc9eb8"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.517957 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.531415 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.533639 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.533809 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.534080 3549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:03.534380 3549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:03.534436 3549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.534477 3549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:03.534504 3549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.540159 3549 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:39861
I20260927 06:59:03.540233 3677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:39861 every 8 connection(s)
I20260927 06:59:03.540536 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:03.546080 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3549
I20260927 06:59:03.546159 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:03.546327 3300 external_mini_cluster.cc:1428] Setting key c46a957d77b860f910408187cf5016af
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:03.550148 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.541247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:34571 (local address 127.3.57.62:34397)
0927 06:59:03.541292 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.541295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.541534 (+ 239us) server_negotiation.cc:407] Connection header received
0927 06:59:03.541975 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.541976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.541986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.542039 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.542932 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.543419 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.544498 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.544678 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.544730 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.547073 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.547074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.547074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.547080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.548789 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.549168 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.549170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.549170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.549241 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.549491 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.549492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.549492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.549607 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.549682 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.549964 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.550045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 06:59:03.550345 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.541250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34397 (local address 127.3.57.2:34571)
0927 06:59:03.541414 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:03.541448 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:03.541896 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:03.542091 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:03.542094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:03.542657 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:03.542855 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.542859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.543520 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.543521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.544420 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.544422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.544750 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.544751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.544851 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.545406 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:03.545414 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:03.546991 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:03.548883 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.548888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.548890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.549097 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.549320 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.549321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.549321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.549415 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.549683 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:03.549690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:03.549823 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 06:59:03.549931 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:59:03.550083 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260927 06:59:03.550803 3678 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34397
I20260927 06:59:03.550906 3678 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:03.551097 3678 heartbeater.cc:511] Master 127.3.57.62:34397 requested a full tablet report, sending...
I20260927 06:59:03.551523 3352 ts_manager.cc:194] Registered new tserver with Master: 14097b57be0c4d18bbcdee3a8cf84396 (127.3.57.2:39861)
I20260927 06:59:03.551921 3352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:34571
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:03.570477 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:03.650628 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:03.650807 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:03.650828 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:03.651952 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:03.651988 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:03.652014 3683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:03.653370 3683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:03.653578 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:03.653803 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:03.658829 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:03.658844 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.658948 3683 server_base.cc:1034] running on GCE node
W20260927 06:59:03.658838 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:03.659168 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:03.659413 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:03.660561 3683 hybrid_clock.cc:648] HybridClock initialized: now 1790492343660560 us; error 34 us; skew 500 ppm
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:03.662914 3683 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:03.663319 3683 webserver.cc:460] Webserver started at http://127.3.57.3:34191/ using document root <none> and password file <none>
I20260927 06:59:03.663483 3683 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:03.663522 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:03.663663 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:03.664611 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "9ba38e6e9632404bae8268c9dd2b1815"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "9afbceaa1bd5a244fc36bf1cf27de265"
server_key_iv: "627e20dbf7a44f08ff2df059fc8a5581"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.664893 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "9ba38e6e9632404bae8268c9dd2b1815"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "9afbceaa1bd5a244fc36bf1cf27de265"
server_key_iv: "627e20dbf7a44f08ff2df059fc8a5581"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.666186 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:03.666869 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.667083 3683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:03.667172 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "9ba38e6e9632404bae8268c9dd2b1815"
format_stamp: "Formatted at 2026-09-27 06:59:03 on dist-test-slave-mk3h"
server_key: "9afbceaa1bd5a244fc36bf1cf27de265"
server_key_iv: "627e20dbf7a44f08ff2df059fc8a5581"
server_key_version: "encryptionkey@0"
I20260927 06:59:03.667227 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:03.684757 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:03.686412 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:03.686517 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:03.686674 3683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:03.686976 3683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:03.687009 3683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.687031 3683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:03.687081 3683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:03.692575 3683 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33489
I20260927 06:59:03.692684 3811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33489 every 8 connection(s)
I20260927 06:59:03.692967 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:03.694677 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3683
I20260927 06:59:03.694757 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:03.694919 3300 external_mini_cluster.cc:1428] Setting key b0d1e48031ff886ed61c9536d857c84f
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:03.701679 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.693663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:46171 (local address 127.3.57.62:34397)
0927 06:59:03.693721 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.693723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.693898 (+ 175us) server_negotiation.cc:407] Connection header received
0927 06:59:03.694337 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.694338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.694347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.694396 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.695478 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.696217 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.697144 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.697328 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.697377 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.699257 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.699258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.699259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.699265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.700633 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.700968 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.700969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.700969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.701015 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.701212 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.701212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.701213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.701306 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.701355 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.701566 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.701620 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 06:59:03.701948 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.693646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34397 (local address 127.3.57.3:46171)
0927 06:59:03.693783 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:03.693814 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:03.694237 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:03.694448 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:03.694451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:03.695121 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:03.695374 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.695379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.696307 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.696308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.697073 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:03.697075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.697395 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.697396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:03.697489 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.697896 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:03.697903 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:03.699181 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:03.700700 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.700705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.700707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.700904 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.701070 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:03.701070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:03.701071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:03.701159 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:03.701359 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:03.701362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:03.701462 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 06:59:03.701539 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 06:59:03.701645 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260927 06:59:03.702371 3812 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34397
I20260927 06:59:03.702472 3812 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:03.702651 3812 heartbeater.cc:511] Master 127.3.57.62:34397 requested a full tablet report, sending...
I20260927 06:59:03.703001 3352 ts_manager.cc:194] Registered new tserver with Master: 9ba38e6e9632404bae8268c9dd2b1815 (127.3.57.3:33489)
I20260927 06:59:03.703383 3352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:46171
I20260927 06:59:03.706949 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:03.728920 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.721552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.3.57.1:33591)
0927 06:59:03.721761 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.721772 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.721782 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:03.721806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.721808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.721821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.721887 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.722447 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.723156 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.723972 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.724274 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.724328 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.726368 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.726376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.726376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.726382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.727888 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.728222 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.728224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.728225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.728275 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.728440 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.728440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.728441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.728553 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.728650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.728725 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.728791 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
W20260927 06:59:03.729329 3458 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50434
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:03.752516 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:03.745976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.3.57.1:33591)
0927 06:59:03.746044 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:03.746046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:03.746110 (+ 64us) server_negotiation.cc:407] Connection header received
0927 06:59:03.746193 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:03.746194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:03.746206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:03.746256 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:03.746793 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.747280 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.748115 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:03.748335 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:03.748422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:03.750461 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:03.750462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:03.750462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:03.750468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:03.751686 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.751974 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.751974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.751975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.752023 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:03.752166 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:03.752166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:03.752167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:03.752242 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:03.752323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:03.752397 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 06:59:03.752453 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 06:59:03.753058 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3415
I20260927 06:59:03.758548 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3549
I20260927 06:59:03.763954 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3683
I20260927 06:59:03.768129 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3320
2026-09-27T06:59:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1802 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3826](info): set up 2 sockets
Sep 27 06:59:03 dist-test-slave-mk3h krb5kdc[3826](info): commencing operation
krb5kdc: starting...
W20260927 06:59:04.823110 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:04 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:04.938651 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41867
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:05.016618 3839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:05.016841 3839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:05.016875 3839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:05.017973 3839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:05.018036 3839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:05.018061 3839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:05.018091 3839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:05.018115 3839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:05.019588 3839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:05.019852 3839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:05.020059 3839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:05.024818 3845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.024857 3839 server_base.cc:1034] running on GCE node
W20260927 06:59:05.024811 3844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:05.024811 3847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.025224 3839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:05.025445 3839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:05.026605 3839 hybrid_clock.cc:648] HybridClock initialized: now 1790492345026591 us; error 29 us; skew 500 ppm
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:05.029043 3839 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:05.029462 3839 webserver.cc:460] Webserver started at http://127.3.57.62:41501/ using document root <none> and password file <none>
I20260927 06:59:05.029642 3839 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:05.029681 3839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:05.029776 3839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:05.030692 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "c3c7ae0031ad422ea71e625703b724dd"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "46783dc7a29ee988edcef5418b40554f"
server_key_iv: "0701bc4e53bc130c915a0816a123d86d"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.030987 3839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "c3c7ae0031ad422ea71e625703b724dd"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "46783dc7a29ee988edcef5418b40554f"
server_key_iv: "0701bc4e53bc130c915a0816a123d86d"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.032292 3839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:05.032944 3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.033234 3839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:05.033317 3839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "c3c7ae0031ad422ea71e625703b724dd"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "46783dc7a29ee988edcef5418b40554f"
server_key_iv: "0701bc4e53bc130c915a0816a123d86d"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.033373 3839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:05.049572 3839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:05.051246 3839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:05.051347 3839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:05.055052 3839 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41867
I20260927 06:59:05.055114 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41867 every 8 connection(s)
I20260927 06:59:05.055435 3839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:05.056304 3907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:05.058416 3907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd: Bootstrap starting.
I20260927 06:59:05.058988 3907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:05.059278 3907 log.cc:824] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:05.059985 3907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd: No bootstrap required, opened a new log
I20260927 06:59:05.061530 3907 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } }
I20260927 06:59:05.061685 3907 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:05.061770 3907 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c7ae0031ad422ea71e625703b724dd, State: Initialized, Role: FOLLOWER
I20260927 06:59:05.061936 3907 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } }
I20260927 06:59:05.062023 3907 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:05.062058 3907 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:05.062108 3907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:05.062601 3907 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } }
I20260927 06:59:05.062721 3907 leader_election.cc:302] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c7ae0031ad422ea71e625703b724dd; no voters:
I20260927 06:59:05.062879 3907 leader_election.cc:288] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:05.062912 3910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:05.062991 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3839
I20260927 06:59:05.063084 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:05.063154 3907 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:05.063246 3300 external_mini_cluster.cc:1428] Setting key 6c5217ed88b4c3a2c7e4df6ba16a7f65
I20260927 06:59:05.063238 3910 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [term 1 LEADER]: Becoming Leader. State: Replica: c3c7ae0031ad422ea71e625703b724dd, State: Running, Role: LEADER
I20260927 06:59:05.063320 3910 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } }
I20260927 06:59:05.063727 3912 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c7ae0031ad422ea71e625703b724dd. Latest consensus state: current_term: 1 leader_uuid: "c3c7ae0031ad422ea71e625703b724dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } } }
I20260927 06:59:05.063728 3911 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c7ae0031ad422ea71e625703b724dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c7ae0031ad422ea71e625703b724dd" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41867 } } }
I20260927 06:59:05.063926 3912 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:05.063946 3911 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:05.064251 3921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:05.064771 3921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:05.066382 3921 catalog_manager.cc:1347] Generated new cluster ID: 2b6346d3f5584f9fbc7e4acdbcde98fe
I20260927 06:59:05.066435 3921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:05.075439 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.3.57.62:41867)
0927 06:59:05.064948 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.064957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.064986 (+ 29us) server_negotiation.cc:407] Connection header received
0927 06:59:05.065097 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.065104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.065157 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.065296 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.065977 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.067313 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.068384 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.069022 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.069143 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.071925 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.071928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.071929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.071940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.073839 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.074247 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.074250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.074250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.074349 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.074566 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.074566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.074567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.074850 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.074990 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.075089 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.075210 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":89,"threads_started":1}
I20260927 06:59:05.078408 3921 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:05.078889 3921 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:05.085122 3921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c7ae0031ad422ea71e625703b724dd: Generated new TSK 0
I20260927 06:59:05.085284 3921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:05.114252 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:05.198625 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:05.198909 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:05.198951 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:05.200578 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:05.200628 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:05.200668 3933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:05.202610 3933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:05.202847 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:05.203094 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:05.208977 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:05.209089 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.209236 3933 server_base.cc:1034] running on GCE node
W20260927 06:59:05.208983 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.209534 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:05.209812 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:05.210937 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790492345210918 us; error 34 us; skew 500 ppm
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:05.214102 3933 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:05.214603 3933 webserver.cc:460] Webserver started at http://127.3.57.1:45171/ using document root <none> and password file <none>
I20260927 06:59:05.214857 3933 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:05.214941 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:05.215075 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:05.216676 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "9438e20c64c5439f83d8f66ad32c20c9"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "33f6fac8548b28195037a4a721a263ed"
server_key_iv: "92b20ce5c88d82eae3e39484129507fa"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.217083 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "9438e20c64c5439f83d8f66ad32c20c9"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "33f6fac8548b28195037a4a721a263ed"
server_key_iv: "92b20ce5c88d82eae3e39484129507fa"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.218495 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:05.219235 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.219419 3933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:05.219527 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "9438e20c64c5439f83d8f66ad32c20c9"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "33f6fac8548b28195037a4a721a263ed"
server_key_iv: "92b20ce5c88d82eae3e39484129507fa"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.219632 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:05.235297 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:05.237805 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:05.237958 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:05.238168 3933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:05.238497 3933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:05.238549 3933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.238590 3933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:05.238615 3933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.244479 3933 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:45285
I20260927 06:59:05.244513 4061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:45285 every 8 connection(s)
I20260927 06:59:05.244875 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:05.248656 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 3933
I20260927 06:59:05.248747 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:05.248919 3300 external_mini_cluster.cc:1428] Setting key 19dcd0e27ea102337a1d8e8d0b8849c7
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:05.254390 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.245608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:35125 (local address 127.3.57.62:41867)
0927 06:59:05.245678 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.245680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.245837 (+ 157us) server_negotiation.cc:407] Connection header received
0927 06:59:05.246684 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.246684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.246696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.246748 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.247271 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.247871 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.249072 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.249335 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.249385 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.251659 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.251661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.251661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.251673 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.253035 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.253455 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.253456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.253456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.253513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.253760 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.253761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.253762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.253871 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.253942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.254217 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.254305 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 06:59:05.254544 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.245585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41867 (local address 127.3.57.1:35125)
0927 06:59:05.245724 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.245761 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.246599 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.246796 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.246799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.247015 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:05.247203 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.247207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.247949 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.247950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.248995 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.248998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.249412 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.249413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.249546 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.249993 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:05.250002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:05.251520 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:05.253137 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.253144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.253146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.253370 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.253579 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.253580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.253580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.253692 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.253945 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:05.253951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:05.254079 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.254182 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.254328 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260927 06:59:05.255045 4062 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41867
I20260927 06:59:05.255148 4062 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:05.255348 4062 heartbeater.cc:511] Master 127.3.57.62:41867 requested a full tablet report, sending...
I20260927 06:59:05.255872 3871 ts_manager.cc:194] Registered new tserver with Master: 9438e20c64c5439f83d8f66ad32c20c9 (127.3.57.1:45285)
I20260927 06:59:05.256510 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:35125
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:05.273842 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:05.352373 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:05.352584 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:05.352619 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:05.353732 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:05.353799 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:05.353844 4067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:05.355322 4067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:05.355597 4067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:05.355815 4067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:05.360585 4072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:05.360589 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.360606 4067 server_base.cc:1034] running on GCE node
W20260927 06:59:05.360579 4073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.360991 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:05.361274 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:05.362462 4067 hybrid_clock.cc:648] HybridClock initialized: now 1790492345362447 us; error 25 us; skew 500 ppm
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:05.365034 4067 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:05.365487 4067 webserver.cc:460] Webserver started at http://127.3.57.2:34957/ using document root <none> and password file <none>
I20260927 06:59:05.365701 4067 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:05.365764 4067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:05.365912 4067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:05.366884 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "d45a75bbacec458c9a8bf3e9aa1f6918"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0e2edab5a04fda2d81e2516f975c5c52"
server_key_iv: "87872c84d9fca15f0234361abc5cf99e"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.367268 4067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "d45a75bbacec458c9a8bf3e9aa1f6918"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0e2edab5a04fda2d81e2516f975c5c52"
server_key_iv: "87872c84d9fca15f0234361abc5cf99e"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.368580 4067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:05.369319 4082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.369485 4067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:05.369578 4067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "d45a75bbacec458c9a8bf3e9aa1f6918"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0e2edab5a04fda2d81e2516f975c5c52"
server_key_iv: "87872c84d9fca15f0234361abc5cf99e"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.369657 4067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:05.381042 4067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:05.383203 4067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:05.383355 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:05.383632 4067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:05.383920 4067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:05.383973 4067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.384004 4067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:05.384032 4067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.389784 4067 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46067
I20260927 06:59:05.389838 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46067 every 8 connection(s)
I20260927 06:59:05.390110 4067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:05.397867 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4067
I20260927 06:59:05.397960 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:05.398135 3300 external_mini_cluster.cc:1428] Setting key 2404f09f8a65f007abc87b45bd767678
I20260927 06:59:05.398792 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.390729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44239 (local address 127.3.57.62:41867)
0927 06:59:05.390816 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.390818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.390971 (+ 153us) server_negotiation.cc:407] Connection header received
0927 06:59:05.391906 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.391907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.391917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.391969 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.392484 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.392980 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.393934 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.394109 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.394158 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.396201 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.396203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.396203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.396210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.397338 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.397679 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.397680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.397680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.397728 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.398032 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.398033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.398033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.398184 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.398236 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.398598 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.398687 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:05.398911 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.390698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41867 (local address 127.3.57.2:44239)
0927 06:59:05.390852 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.390883 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.391810 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.392019 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.392022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.392244 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:05.392416 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.392420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.393072 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.393073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.393863 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.393865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.394175 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.394176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.394265 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.394646 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:05.394653 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:05.396120 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:05.397406 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.397411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.397412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.397617 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.397791 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.397792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.397792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.397931 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.398292 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:05.398298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:05.398464 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.398557 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.398669 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260927 06:59:05.399403 4196 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41867
I20260927 06:59:05.399498 4196 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:05.399701 4196 heartbeater.cc:511] Master 127.3.57.62:41867 requested a full tablet report, sending...
I20260927 06:59:05.400077 3871 ts_manager.cc:194] Registered new tserver with Master: d45a75bbacec458c9a8bf3e9aa1f6918 (127.3.57.2:46067)
I20260927 06:59:05.400558 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:44239
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:05.441488 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:05.545145 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:05.545430 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:05.545478 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:05.546976 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:05.547065 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:05.547112 4201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:05.549186 4201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:05.549537 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:05.549800 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:05.556208 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:05.556164 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:05.557641 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:05.558733 4201 server_base.cc:1034] running on GCE node
I20260927 06:59:05.559434 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:05.559834 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:05.561235 4201 hybrid_clock.cc:648] HybridClock initialized: now 1790492345561216 us; error 37 us; skew 500 ppm
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:05.564412 4201 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:05.565079 4201 webserver.cc:460] Webserver started at http://127.3.57.3:40203/ using document root <none> and password file <none>
I20260927 06:59:05.565338 4201 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:05.565407 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:05.565562 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:05.566675 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "7c38b2f73fca40ac8606f60cf2543320"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0b22d57d32f33ee45fef976b648e28a1"
server_key_iv: "d14365774cad0fac3d24a7fdcf706778"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.567070 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "7c38b2f73fca40ac8606f60cf2543320"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0b22d57d32f33ee45fef976b648e28a1"
server_key_iv: "d14365774cad0fac3d24a7fdcf706778"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.568619 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:05.569617 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.569933 4201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:05.570161 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "7c38b2f73fca40ac8606f60cf2543320"
format_stamp: "Formatted at 2026-09-27 06:59:05 on dist-test-slave-mk3h"
server_key: "0b22d57d32f33ee45fef976b648e28a1"
server_key_iv: "d14365774cad0fac3d24a7fdcf706778"
server_key_version: "encryptionkey@0"
I20260927 06:59:05.570250 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:05.588111 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:05.590443 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:05.590714 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:05.590975 4201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:05.591301 4201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:05.591360 4201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.591434 4201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:05.591465 4201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:05.604859 4201 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:36417
I20260927 06:59:05.604923 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:36417 every 8 connection(s)
I20260927 06:59:05.605319 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:05.611043 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4201
I20260927 06:59:05.611169 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:05.611340 3300 external_mini_cluster.cc:1428] Setting key 2108ff5718d914ce75c5bd414ea4028b
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:05.621963 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.606305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41867 (local address 127.3.57.3:57265)
0927 06:59:05.606523 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.606580 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.607868 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.608247 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.608251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.608655 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:05.608981 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.608988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.609778 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.609779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.610635 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.610637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.611169 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.611170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.611318 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.612079 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:05.612093 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:05.613808 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:05.618199 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.618210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.618212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.618473 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.620551 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:05.620552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:05.620552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:05.620711 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:05.621054 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:05.621059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:05.621222 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.621315 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.621588 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260927 06:59:05.621997 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.606347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:57265 (local address 127.3.57.62:41867)
0927 06:59:05.606431 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.606434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.606736 (+ 302us) server_negotiation.cc:407] Connection header received
0927 06:59:05.608006 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.608008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.608024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.608107 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.609128 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.609650 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.610757 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.610962 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.611165 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.614860 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.614862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.614863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.614872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.618045 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.620340 (+ 2295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.620342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.620343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.620421 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.620818 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.620819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.620820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.620944 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.621001 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.621861 (+ 860us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.621934 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 06:59:05.623257 4330 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41867
I20260927 06:59:05.623417 4330 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:05.623690 4330 heartbeater.cc:511] Master 127.3.57.62:41867 requested a full tablet report, sending...
I20260927 06:59:05.624117 3871 ts_manager.cc:194] Registered new tserver with Master: 7c38b2f73fca40ac8606f60cf2543320 (127.3.57.3:36417)
I20260927 06:59:05.624513 3871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:57265
I20260927 06:59:05.635459 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:05.645113 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.637878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.3.57.62:41867)
0927 06:59:05.637954 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.637955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.637963 (+ 8us) server_negotiation.cc:407] Connection header received
0927 06:59:05.637984 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.637984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.637993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.638037 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.638759 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.639254 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.640350 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.640697 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.640772 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.642222 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.642223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.642224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.642229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.643635 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.644037 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.644038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.644038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.644096 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.644361 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.644362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.644362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.644535 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.644615 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.644836 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.644970 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:59:05.648110 3871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:05.649899 3871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:05.657594 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.653579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.3.57.3:36417)
0927 06:59:05.653756 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.653761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.653770 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:05.653831 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.653833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.653848 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.653926 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:05.654031 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.655235 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.656477 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.657112 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.657225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.657295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.657455 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.657501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260927 06:59:05.657594 4345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.653422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36417 (local address 127.0.0.1:50908)
0927 06:59:05.653543 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.653552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.653591 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.653932 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.653933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.653935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:05.653971 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.653972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.655340 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.655340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.656353 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.656356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.657229 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.657230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.657314 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.657330 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.657438 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.657519 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":88,"thread_start_us":83,"threads_started":1}
I20260927 06:59:05.659945 4264 tablet_service.cc:1467] Processing CreateTablet for tablet 3e036645a6fe425e94560526c1da6c27 (DEFAULT_TABLE table=test-table [id=02f15142de254b85943f4a4514995848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:05.660223 4341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.652969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:45285 (local address 127.0.0.1:42096)
0927 06:59:05.653111 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.653157 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.653224 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.653761 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.653764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.653767 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:05.653902 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.653906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.655274 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.655275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.657525 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.657528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.659898 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.659899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.659991 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.659997 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.660066 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.660125 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":309,"thread_start_us":83,"threads_started":1}
I20260927 06:59:05.660281 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.653244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.3.57.1:45285)
0927 06:59:05.653565 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.653569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.653579 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:05.653615 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.653617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.653630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.653809 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:05.653999 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.655137 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.658050 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.659781 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.659886 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.659961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.660113 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.660172 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
I20260927 06:59:05.660543 4264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e036645a6fe425e94560526c1da6c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:05.661700 4342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.653216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46067 (local address 127.0.0.1:33684)
0927 06:59:05.653388 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.653398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.653442 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.653761 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.653764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.653769 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:05.653889 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.653892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.657262 (+ 3370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.657264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.658856 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.658859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.661405 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.661406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.661495 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.661500 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.661559 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.661613 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260927 06:59:05.661702 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.653359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.3.57.2:46067)
0927 06:59:05.653550 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.653555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.653567 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:05.653611 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.653613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.653630 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.653980 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:05.654064 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.656639 (+ 2575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.659823 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.661277 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.661400 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.661474 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.661577 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.661624 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260927 06:59:05.662792 3996 tablet_service.cc:1467] Processing CreateTablet for tablet 3e036645a6fe425e94560526c1da6c27 (DEFAULT_TABLE table=test-table [id=02f15142de254b85943f4a4514995848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:05.662827 4130 tablet_service.cc:1467] Processing CreateTablet for tablet 3e036645a6fe425e94560526c1da6c27 (DEFAULT_TABLE table=test-table [id=02f15142de254b85943f4a4514995848]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:05.663224 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e036645a6fe425e94560526c1da6c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:05.663259 4130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e036645a6fe425e94560526c1da6c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:05.666136 4347 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Bootstrap starting.
I20260927 06:59:05.666640 4348 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Bootstrap starting.
I20260927 06:59:05.666826 4349 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: Bootstrap starting.
I20260927 06:59:05.666993 4347 tablet_bootstrap.cc:654] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:05.667318 4348 tablet_bootstrap.cc:654] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:05.667519 4349 tablet_bootstrap.cc:654] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:05.667907 4347 log.cc:824] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:05.667965 4348 log.cc:824] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:05.668110 4349 log.cc:824] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:05.668740 4348 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: No bootstrap required, opened a new log
I20260927 06:59:05.668831 4348 ts_tablet_manager.cc:1397] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:05.668874 4349 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: No bootstrap required, opened a new log
I20260927 06:59:05.668932 4349 ts_tablet_manager.cc:1397] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:05.670307 4348 raft_consensus.cc:348] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.670321 4349 raft_consensus.cc:348] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.670478 4349 raft_consensus.cc:374] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:05.670481 4348 raft_consensus.cc:374] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:05.670531 4349 raft_consensus.cc:729] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45a75bbacec458c9a8bf3e9aa1f6918, State: Initialized, Role: FOLLOWER
I20260927 06:59:05.670531 4348 raft_consensus.cc:729] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9438e20c64c5439f83d8f66ad32c20c9, State: Initialized, Role: FOLLOWER
I20260927 06:59:05.670663 4349 consensus_queue.cc:260] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.670671 4348 consensus_queue.cc:260] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.670994 4348 ts_tablet_manager.cc:1428] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:05.671011 4349 ts_tablet_manager.cc:1428] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:05.671018 4062 heartbeater.cc:503] Master 127.3.57.62:41867 was elected leader, sending a full tablet report...
I20260927 06:59:05.671105 4196 heartbeater.cc:503] Master 127.3.57.62:41867 was elected leader, sending a full tablet report...
I20260927 06:59:05.671463 4347 tablet_bootstrap.cc:492] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: No bootstrap required, opened a new log
I20260927 06:59:05.671566 4347 ts_tablet_manager.cc:1397] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 06:59:05.674280 4347 raft_consensus.cc:348] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.674448 4347 raft_consensus.cc:374] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:05.674470 4347 raft_consensus.cc:729] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c38b2f73fca40ac8606f60cf2543320, State: Initialized, Role: FOLLOWER
I20260927 06:59:05.674607 4347 consensus_queue.cc:260] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.674930 4347 ts_tablet_manager.cc:1428] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:05.675385 4330 heartbeater.cc:503] Master 127.3.57.62:41867 was elected leader, sending a full tablet report...
W20260927 06:59:05.747069 4063 tablet.cc:2369] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:05.782435 4355 raft_consensus.cc:482] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:05.782678 4355 raft_consensus.cc:504] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.783063 4355 leader_election.cc:288] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9438e20c64c5439f83d8f66ad32c20c9 (127.3.57.1:45285), d45a75bbacec458c9a8bf3e9aa1f6918 (127.3.57.2:46067)
I20260927 06:59:05.786729 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.783373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:45285 (local address 127.3.57.3:59717)
0927 06:59:05.783511 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.783526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.783614 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.783830 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.783831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.783833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:05.783911 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.783913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.784820 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.784822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.785661 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.785664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.786469 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.786470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.786555 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.786559 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.786616 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.786661 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":22}
I20260927 06:59:05.786818 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.783522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:59717 (local address 127.3.57.1:45285)
0927 06:59:05.783628 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.783630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.783642 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:05.783675 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.783677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.783690 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.783765 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:05.784094 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.784631 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.785859 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.786288 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.786345 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.786355 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.786666 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.786713 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 06:59:05.786993 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.783640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:46783 (local address 127.3.57.2:46067)
0927 06:59:05.783694 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.783696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.783812 (+ 116us) server_negotiation.cc:407] Connection header received
0927 06:59:05.783839 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.783840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.783852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.783915 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:05.784268 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.784892 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.786038 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.786551 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.786625 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.786634 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.786854 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.786924 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 06:59:05.787185 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.783525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46067 (local address 127.3.57.3:46783)
0927 06:59:05.783707 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:05.783717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:05.783774 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:05.784094 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:05.784095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:05.784097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:05.784160 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.784162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.785014 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.785014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.785913 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:05.785915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.786670 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.786671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:05.786744 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.786748 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:05.787073 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 06:59:05.787114 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260927 06:59:05.787155 4016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e036645a6fe425e94560526c1da6c27" candidate_uuid: "7c38b2f73fca40ac8606f60cf2543320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9438e20c64c5439f83d8f66ad32c20c9" is_pre_election: true
I20260927 06:59:05.787361 4016 raft_consensus.cc:2454] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c38b2f73fca40ac8606f60cf2543320 in term 0.
I20260927 06:59:05.787650 4220 leader_election.cc:302] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c38b2f73fca40ac8606f60cf2543320, 9438e20c64c5439f83d8f66ad32c20c9; no voters:
I20260927 06:59:05.787644 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e036645a6fe425e94560526c1da6c27" candidate_uuid: "7c38b2f73fca40ac8606f60cf2543320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" is_pre_election: true
I20260927 06:59:05.787806 4150 raft_consensus.cc:2454] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c38b2f73fca40ac8606f60cf2543320 in term 0.
I20260927 06:59:05.787812 4355 raft_consensus.cc:2781] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:05.787868 4355 raft_consensus.cc:482] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:05.787895 4355 raft_consensus.cc:3037] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:05.788661 4355 raft_consensus.cc:504] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.788812 4355 leader_election.cc:288] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [CANDIDATE]: Term 1 election: Requested vote from peers 9438e20c64c5439f83d8f66ad32c20c9 (127.3.57.1:45285), d45a75bbacec458c9a8bf3e9aa1f6918 (127.3.57.2:46067)
I20260927 06:59:05.789027 4150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e036645a6fe425e94560526c1da6c27" candidate_uuid: "7c38b2f73fca40ac8606f60cf2543320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918"
I20260927 06:59:05.789023 4016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e036645a6fe425e94560526c1da6c27" candidate_uuid: "7c38b2f73fca40ac8606f60cf2543320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9438e20c64c5439f83d8f66ad32c20c9"
I20260927 06:59:05.789116 4150 raft_consensus.cc:3037] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:05.789116 4016 raft_consensus.cc:3037] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:05.789989 4150 raft_consensus.cc:2454] T 3e036645a6fe425e94560526c1da6c27 P d45a75bbacec458c9a8bf3e9aa1f6918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c38b2f73fca40ac8606f60cf2543320 in term 1.
I20260927 06:59:05.789984 4016 raft_consensus.cc:2454] T 3e036645a6fe425e94560526c1da6c27 P 9438e20c64c5439f83d8f66ad32c20c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c38b2f73fca40ac8606f60cf2543320 in term 1.
I20260927 06:59:05.790171 4220 leader_election.cc:302] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c38b2f73fca40ac8606f60cf2543320, d45a75bbacec458c9a8bf3e9aa1f6918; no voters:
I20260927 06:59:05.790294 4355 raft_consensus.cc:2781] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:05.790482 4355 raft_consensus.cc:686] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [term 1 LEADER]: Becoming Leader. State: Replica: 7c38b2f73fca40ac8606f60cf2543320, State: Running, Role: LEADER
I20260927 06:59:05.790591 4355 consensus_queue.cc:237] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } }
I20260927 06:59:05.791930 3870 catalog_manager.cc:5495] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c38b2f73fca40ac8606f60cf2543320 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "7c38b2f73fca40ac8606f60cf2543320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9438e20c64c5439f83d8f66ad32c20c9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45a75bbacec458c9a8bf3e9aa1f6918" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c38b2f73fca40ac8606f60cf2543320" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36417 } health_report { overall_health: HEALTHY } } }
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:05.814380 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.805392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.3.57.1:45285)
0927 06:59:05.805441 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.805443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.805569 (+ 126us) server_negotiation.cc:407] Connection header received
0927 06:59:05.805654 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.805655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.805669 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.805735 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.806300 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.806798 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.807517 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.807761 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.807860 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.810589 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.810600 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.810600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.810607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.812379 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.812847 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.812850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.812850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.812919 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.813487 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.813489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.813489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.813669 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.813803 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.814229 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.814306 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:05.841152 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.833522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.3.57.1:45285)
0927 06:59:05.833577 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.833579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.833680 (+ 101us) server_negotiation.cc:407] Connection header received
0927 06:59:05.833727 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.833728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.833737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.833810 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.834331 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.834821 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.835517 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.835762 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.835831 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.838191 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.838192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.838192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.838198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.839891 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.840281 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.840282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.840282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.840363 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.840563 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.840564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.840565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.840679 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.840787 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.840965 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.841047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 06:59:05.856588 4331 tablet.cc:2369] T 3e036645a6fe425e94560526c1da6c27 P 7c38b2f73fca40ac8606f60cf2543320: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[3826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:05.862694 4343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:05.855987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.3.57.1:45285)
0927 06:59:05.856071 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:05.856073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:05.856096 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:05.856159 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:05.856160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:05.856169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:05.856213 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:05.856816 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.857281 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.857988 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:05.858158 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:05.858205 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:05.860358 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:05.860359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:05.860360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:05.860367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:05.861762 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.862096 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.862097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.862097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.862144 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:05.862299 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:05.862300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:05.862300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:05.862384 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:05.862451 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:05.862534 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 06:59:05.862594 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 06:59:05.863233 3976 tablet_service.cc:2908] Rejecting scan request for tablet 3e036645a6fe425e94560526c1da6c27: Uninitialized: clean time has not yet been initialized
I20260927 06:59:05.864140 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3933
I20260927 06:59:05.869220 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4067
I20260927 06:59:05.873840 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4201
I20260927 06:59:05.878084 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 3839
2026-09-27T06:59:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2111 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[4371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[4371](info): set up 2 sockets
Sep 27 06:59:05 dist-test-slave-mk3h krb5kdc[4371](info): commencing operation
krb5kdc: starting...
W20260927 06:59:06.915289 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:06 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:07.031775 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45333
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:45333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:07.109048 4384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:07.109229 4384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:07.109251 4384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:07.110390 4384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:07.110430 4384 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 06:59:07.110445 4384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:07.110482 4384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:07.110512 4384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:07.110528 4384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:07.111974 4384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:45333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:07.112205 4384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:07.112426 4384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:07.117086 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.117108 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.117125 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.117362 4384 server_base.cc:1034] running on GCE node
I20260927 06:59:07.117594 4384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:07.117877 4384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:07.119037 4384 hybrid_clock.cc:648] HybridClock initialized: now 1790492347119019 us; error 31 us; skew 500 ppm
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:07.125185 4384 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:07.125677 4384 webserver.cc:460] Webserver started at http://127.3.57.62:40253/ using document root <none> and password file <none>
I20260927 06:59:07.125919 4384 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:07.125988 4384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:07.126132 4384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:07.127161 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "1729dd50439344b88931e5347597523b"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "f398799679e5e8c5db619eaf4e22b8b3"
server_key_iv: "9f3e85bcb12a9005251c854e4baad756"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.127528 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "1729dd50439344b88931e5347597523b"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "f398799679e5e8c5db619eaf4e22b8b3"
server_key_iv: "9f3e85bcb12a9005251c854e4baad756"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.128841 4384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:07.129516 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.129681 4384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:07.129781 4384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "1729dd50439344b88931e5347597523b"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "f398799679e5e8c5db619eaf4e22b8b3"
server_key_iv: "9f3e85bcb12a9005251c854e4baad756"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.129853 4384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:07.137315 4384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:07.139058 4384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:07.139212 4384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.143690 4384 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:45333
I20260927 06:59:07.143743 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:45333 every 8 connection(s)
I20260927 06:59:07.144073 4384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:07.144722 4462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.145797 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4384
I20260927 06:59:07.145906 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:07.146051 3300 external_mini_cluster.cc:1428] Setting key d9b253bc53cfc2eff14bb48564089299
I20260927 06:59:07.147135 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Bootstrap starting.
I20260927 06:59:07.147754 4462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.148030 4462 log.cc:824] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:07.148679 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: No bootstrap required, opened a new log
I20260927 06:59:07.150559 4462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } }
I20260927 06:59:07.150749 4462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.150792 4462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1729dd50439344b88931e5347597523b, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.150926 4462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } }
I20260927 06:59:07.150995 4462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:07.151015 4462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:07.151090 4462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:07.151652 4462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } }
I20260927 06:59:07.151737 4462 leader_election.cc:302] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1729dd50439344b88931e5347597523b; no voters:
I20260927 06:59:07.151911 4462 leader_election.cc:288] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:07.151968 4467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:07.152143 4467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [term 1 LEADER]: Becoming Leader. State: Replica: 1729dd50439344b88931e5347597523b, State: Running, Role: LEADER
I20260927 06:59:07.152155 4462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:07.152227 4467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } }
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:07.152527 4467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1729dd50439344b88931e5347597523b. Latest consensus state: current_term: 1 leader_uuid: "1729dd50439344b88931e5347597523b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } } }
I20260927 06:59:07.152623 4467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:07.152539 4468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1729dd50439344b88931e5347597523b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1729dd50439344b88931e5347597523b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45333 } } }
I20260927 06:59:07.152818 4468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:07.152925 4474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:07.153542 4474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:07.155247 4474 catalog_manager.cc:1347] Generated new cluster ID: 8f7d8b8d90b5426bbdeca7d54efdf8c6
I20260927 06:59:07.155308 4474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:07.156714 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.146539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.3.57.62:45333)
0927 06:59:07.146696 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.146703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.146727 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:07.147114 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.147119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.147160 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.147330 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.148028 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.148977 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.149857 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.150295 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.150407 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.153168 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.153172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.153173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.153185 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.155149 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.155541 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.155561 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.155561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.155667 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.155893 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.155894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.155894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.156159 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.156288 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.156419 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.156539 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260927 06:59:07.161190 4474 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:07.161664 4474 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:07.172973 4474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Generated new TSK 0
I20260927 06:59:07.173151 4474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:07.204712 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45333
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:07.283370 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:07.283696 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:07.283767 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:07.285295 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:07.285356 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:07.285394 4488 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:07.285430 4488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:07.286810 4488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:45333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:07.287055 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:07.287292 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:07.292106 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.292101 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.292256 4488 server_base.cc:1034] running on GCE node
W20260927 06:59:07.292162 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.292490 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:07.292678 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:07.293848 4488 hybrid_clock.cc:648] HybridClock initialized: now 1790492347293827 us; error 33 us; skew 500 ppm
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:07.296396 4488 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:07.296859 4488 webserver.cc:460] Webserver started at http://127.3.57.1:39025/ using document root <none> and password file <none>
I20260927 06:59:07.297078 4488 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:07.297159 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:07.297305 4488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:07.298326 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "2f2e59e1dd504615939cf049822899c4"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "67a2f25211d3d3af3f5d83d339eeee8a"
server_key_iv: "5bedfd2bdd282f0f6297a5f014eae633"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.298738 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "2f2e59e1dd504615939cf049822899c4"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "67a2f25211d3d3af3f5d83d339eeee8a"
server_key_iv: "5bedfd2bdd282f0f6297a5f014eae633"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.300143 4488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:07.300877 4503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.301039 4488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:07.301141 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "2f2e59e1dd504615939cf049822899c4"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "67a2f25211d3d3af3f5d83d339eeee8a"
server_key_iv: "5bedfd2bdd282f0f6297a5f014eae633"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.301213 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:07.318188 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:07.320664 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:07.320817 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.321496 4488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:07.321563 4488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.321605 4488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:07.321635 4488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.328624 4488 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44405
I20260927 06:59:07.328960 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:07.329555 4618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44405 every 8 connection(s)
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:07.335978 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.321919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:41571 (local address 127.3.57.62:45333)
0927 06:59:07.321979 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.321982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.322223 (+ 241us) server_negotiation.cc:407] Connection header received
0927 06:59:07.323022 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.323024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.323038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.323094 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.323984 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.324944 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.326387 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.326692 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.326753 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.333447 (+ 6694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.333448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.333449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.333455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.334897 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.335225 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.335226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.335226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.335271 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.335467 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.335468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.335468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.335608 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.335656 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.335854 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.335919 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 06:59:07.336139 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.321900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.1:41571)
0927 06:59:07.322083 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.322127 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.322903 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.323227 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.323232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.323599 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.323873 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.323880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.325041 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.325041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.326283 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.326286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.326826 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.326827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.326955 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.331895 (+ 4940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.331903 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.333363 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.334968 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.334973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.334974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.335167 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.335334 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.335334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.335335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.335418 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.335649 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.335653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.335753 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.335831 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.335935 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260927 06:59:07.336607 4619 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45333
I20260927 06:59:07.336700 4619 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:07.336876 4619 heartbeater.cc:511] Master 127.3.57.62:45333 requested a full tablet report, sending...
I20260927 06:59:07.337447 4415 ts_manager.cc:194] Registered new tserver with Master: 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405)
I20260927 06:59:07.338359 4415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:41571
I20260927 06:59:07.339151 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4488
I20260927 06:59:07.339258 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:07.339457 3300 external_mini_cluster.cc:1428] Setting key 4d88d8783bf9f9851577a9f913c4c4a0
I20260927 06:59:07.343381 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.337705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:49931 (local address 127.3.57.62:45333)
0927 06:59:07.337743 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.337745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.337827 (+ 82us) server_negotiation.cc:407] Connection header received
0927 06:59:07.337891 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.337892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.337899 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.337937 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.338258 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.338805 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.339756 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.340038 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.340106 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.340909 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.340910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.340910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.340917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.342266 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.342625 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.342626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.342627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.342680 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.342878 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.342878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.342879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.342984 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.343056 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.343225 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.343295 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:07.343403 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.337622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.1:49931)
0927 06:59:07.337772 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.337783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.337840 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.337948 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.337949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.338150 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.338201 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.338202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.338888 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.338890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.339664 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.339666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.340125 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.340126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.340199 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.340581 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.340585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.340826 (+ 241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.342374 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.342375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.342375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.342542 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.342750 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.342751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.342751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.342807 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.343060 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.343061 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.343101 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.343197 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.343294 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":5,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:07.362609 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45333
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:07.441430 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:07.441623 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:07.441648 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:07.442745 4628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:07.442780 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:07.442798 4628 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:07.442816 4628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:07.444209 4628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:45333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:07.444386 4628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:07.444548 4628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:07.449270 4634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.449350 4628 server_base.cc:1034] running on GCE node
W20260927 06:59:07.449297 4633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.449309 4636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.449622 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:07.449847 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:07.451035 4628 hybrid_clock.cc:648] HybridClock initialized: now 1790492347451017 us; error 35 us; skew 500 ppm
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:07.453657 4628 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:07.454099 4628 webserver.cc:460] Webserver started at http://127.3.57.2:34577/ using document root <none> and password file <none>
I20260927 06:59:07.454330 4628 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:07.454373 4628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:07.454511 4628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:07.455536 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "59b496ef67084ca8b8e0b219e3d8ada6"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "e8f259a00ca3813cbdcf486de9186b2c"
server_key_iv: "7acf5b00d969f949a3d39d3acc2a2622"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.455968 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "59b496ef67084ca8b8e0b219e3d8ada6"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "e8f259a00ca3813cbdcf486de9186b2c"
server_key_iv: "7acf5b00d969f949a3d39d3acc2a2622"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.457299 4628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:07.457971 4643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.458144 4628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:07.458228 4628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "59b496ef67084ca8b8e0b219e3d8ada6"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "e8f259a00ca3813cbdcf486de9186b2c"
server_key_iv: "7acf5b00d969f949a3d39d3acc2a2622"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.458305 4628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:07.468441 4628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:07.470041 4628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:07.470201 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.470700 4628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:07.470738 4628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.470759 4628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:07.470772 4628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:07.477897 4628 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:36915
I20260927 06:59:07.477962 4758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:36915 every 8 connection(s)
I20260927 06:59:07.478263 4628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:07.480005 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.471101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44789 (local address 127.3.57.62:45333)
0927 06:59:07.471166 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.471168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.471350 (+ 182us) server_negotiation.cc:407] Connection header received
0927 06:59:07.471849 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.471851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.471864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.471933 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.472575 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.473424 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.474415 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.474659 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.474755 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.477343 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.477345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.477345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.477353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.478903 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.479264 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.479265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.479265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.479321 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.479486 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.479486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.479486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.479635 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.479685 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.479885 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.479944 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 06:59:07.480127 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.471078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.2:44789)
0927 06:59:07.471221 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.471259 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.471756 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.472000 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.472002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.472259 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.472479 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.472484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.473520 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.473521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.474322 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.474324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.474764 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.474765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.474892 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.475387 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.475396 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.477236 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.478972 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.478976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.478977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.479190 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.479360 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.479361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.479361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.479437 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.479679 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.479682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.479777 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.479861 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.479962 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260927 06:59:07.480551 4759 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45333
I20260927 06:59:07.480641 4759 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:07.480824 4759 heartbeater.cc:511] Master 127.3.57.62:45333 requested a full tablet report, sending...
I20260927 06:59:07.481241 4416 ts_manager.cc:194] Registered new tserver with Master: 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2:36915)
I20260927 06:59:07.481724 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:44789
I20260927 06:59:07.486716 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4628
I20260927 06:59:07.486825 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:07.486914 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.481565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:58493 (local address 127.3.57.62:45333)
0927 06:59:07.481601 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.481603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.481723 (+ 120us) server_negotiation.cc:407] Connection header received
0927 06:59:07.481748 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.481748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.481758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.481835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.482137 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.482593 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.483304 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.483489 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.483575 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.484437 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.484439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.484439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.484445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.485780 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.486124 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.486125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.486126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.486181 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.486375 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.486376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.486376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.486516 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.486596 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.486754 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.486826 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 06:59:07.486928 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.481508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.2:58493)
0927 06:59:07.481651 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.481663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.481715 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.481840 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.481841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.482004 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.482053 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.482055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.482680 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.482681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.483200 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.483202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.483583 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.483583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.483658 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.484065 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.484070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.484334 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.485878 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.485879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.485880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.486043 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.486245 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.486246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.486247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.486302 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.486600 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.486601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.486642 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.486727 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.486827 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260927 06:59:07.487010 3300 external_mini_cluster.cc:1428] Setting key c2d8738a2689ab1697e56247c3324106
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:07.514525 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45333
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:07.596100 4768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:07.596300 4768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:07.596323 4768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:07.597482 4768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:07.597522 4768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:07.597543 4768 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:07.597590 4768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:07.599103 4768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:45333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:07.599351 4768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:07.599584 4768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:07.604336 4773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:07.604435 4774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.604548 4768 server_base.cc:1034] running on GCE node
W20260927 06:59:07.604625 4776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:07.604822 4768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:07.605010 4768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:07.606142 4768 hybrid_clock.cc:648] HybridClock initialized: now 1790492347606133 us; error 27 us; skew 500 ppm
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:07.608552 4768 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:07.609000 4768 webserver.cc:460] Webserver started at http://127.3.57.3:33345/ using document root <none> and password file <none>
I20260927 06:59:07.609186 4768 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:07.609225 4768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:07.609313 4768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:07.610281 4768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "c637cee3af7b46b696872d6232b96cf8"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "a1169628766643ce1d788eaf53c2e38e"
server_key_iv: "5bed3b1b6e0664e638ef360a8a13f6fd"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.610570 4768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "c637cee3af7b46b696872d6232b96cf8"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "a1169628766643ce1d788eaf53c2e38e"
server_key_iv: "5bed3b1b6e0664e638ef360a8a13f6fd"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.611816 4768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:07.612476 4783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.612651 4768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:07.612756 4768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "c637cee3af7b46b696872d6232b96cf8"
format_stamp: "Formatted at 2026-09-27 06:59:07 on dist-test-slave-mk3h"
server_key: "a1169628766643ce1d788eaf53c2e38e"
server_key_iv: "5bed3b1b6e0664e638ef360a8a13f6fd"
server_key_version: "encryptionkey@0"
I20260927 06:59:07.612838 4768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:07.631453 4768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:07.633419 4768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:07.633571 4768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:07.634107 4768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:07.634164 4768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.634207 4768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:07.634233 4768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:07.641129 4768 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:42945
I20260927 06:59:07.641204 4898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:42945 every 8 connection(s)
I20260927 06:59:07.641537 4768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:07.643770 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.634507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:40945 (local address 127.3.57.62:45333)
0927 06:59:07.634573 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.634576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.634772 (+ 196us) server_negotiation.cc:407] Connection header received
0927 06:59:07.635241 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.635242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.635255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.635347 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.636021 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.636922 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.638501 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.638842 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.638947 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.641282 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.641284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.641284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.641291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.642615 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.642959 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.642960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.642960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.643008 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.643212 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.643213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.643213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.643322 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.643373 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.643636 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.643708 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 06:59:07.644016 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.634498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.3:40945)
0927 06:59:07.634643 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.634682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.635149 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.635396 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.635399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.635684 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.635919 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.635924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.637016 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.637017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.638408 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.638412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.638995 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.638996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.639147 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.639714 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.639723 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.641198 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.642681 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.642686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.642688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.642894 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.643062 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.643063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.643063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.643158 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.643375 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.643379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.643482 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.643616 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.643752 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260927 06:59:07.644537 4899 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45333
I20260927 06:59:07.644619 4899 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:07.644807 4899 heartbeater.cc:511] Master 127.3.57.62:45333 requested a full tablet report, sending...
I20260927 06:59:07.645227 4415 ts_manager.cc:194] Registered new tserver with Master: c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:07.645756 4415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:40945
I20260927 06:59:07.649170 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4768
I20260927 06:59:07.649281 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:07.649449 3300 external_mini_cluster.cc:1428] Setting key 8b3cbc025c4c69e43752a48579e8c9a4
I20260927 06:59:07.650214 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:07.651116 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.645634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:55643 (local address 127.3.57.62:45333)
0927 06:59:07.645681 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.645683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.645922 (+ 239us) server_negotiation.cc:407] Connection header received
0927 06:59:07.645943 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.645944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.645953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.646016 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.646351 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.646883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.647506 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.647762 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.647814 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.648662 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.648663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.648663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.648668 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.649936 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.650336 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.650337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.650338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.650401 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.650602 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.650604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.650604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.650717 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.650795 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.650954 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.651026 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 06:59:07.651130 4905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.645569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.3.57.3:55643)
0927 06:59:07.645684 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.645694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.645751 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.646036 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.646037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.646216 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:07.646270 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.646271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.646963 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.646963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.647440 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.647441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.647823 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.647824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.647879 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.648313 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:07.648317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:07.648589 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:07.649994 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.649995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.649996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.650223 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.650456 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:07.650457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:07.650457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:07.650535 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:07.650794 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:07.650795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:07.650839 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.650927 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.651027 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:59:07 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:07.672905 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.665944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.3.57.62:45333)
0927 06:59:07.666005 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.666007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.666027 (+ 20us) server_negotiation.cc:407] Connection header received
0927 06:59:07.666053 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.666056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.666069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.666132 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:07.666666 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.667162 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.667946 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.668137 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.668214 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.670331 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:07.670332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:07.670333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:07.670341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:07.671800 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.672143 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.672144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.672144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.672205 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:07.672408 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:07.672409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:07.672410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:07.672519 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:07.672620 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.672728 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.672803 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:59:07.674676 4415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:07.676403 4415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:07.682933 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.678931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36915 (local address 127.0.0.1:50962)
0927 06:59:07.679047 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.679057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.679098 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.679531 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.679532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.679535 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:07.679605 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.679606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680923 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.680924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.681947 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.681950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682610 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.682687 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.682700 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.682796 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.682850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":20,"thread_start_us":72,"threads_started":1}
I20260927 06:59:07.682947 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.679143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.3.57.2:36915)
0927 06:59:07.679310 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.679315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.679327 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:07.679366 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.679368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.679384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.679469 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:07.679688 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680780 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682045 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682511 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.682663 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.682819 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.682858 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260927 06:59:07.683154 4917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.678746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44405 (local address 127.0.0.1:33212)
0927 06:59:07.678907 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.678950 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.679014 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.679536 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.679536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.679538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:07.679605 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.679606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680596 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.680597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.681969 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.681972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682902 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.682972 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.682978 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.683010 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.683061 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":34,"thread_start_us":45,"threads_started":1}
I20260927 06:59:07.683319 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.678757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:42945 (local address 127.0.0.1:34474)
0927 06:59:07.679018 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.679028 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.679080 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.679388 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.679391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.679393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:07.679458 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.679459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680810 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.680811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.681970 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.681972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682900 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.682972 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.683105 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.683134 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.683169 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260927 06:59:07.683539 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.678855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.3.57.1:44405)
0927 06:59:07.679377 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.679381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.679390 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:07.679425 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.679427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.679442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.679692 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:07.679747 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680521 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682095 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682794 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.683289 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.683364 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.683395 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.683442 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":72,"threads_started":1}
I20260927 06:59:07.683802 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.678854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.3.57.3:42945)
0927 06:59:07.679158 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.679162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.679171 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:07.679213 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.679215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.679226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.679539 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:07.679624 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.680734 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.682071 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.682794 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.683468 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.683527 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.683577 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.683709 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":87,"threads_started":1}
I20260927 06:59:07.683837 4693 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 (DEFAULT_TABLE table=test-table [id=abb00550f6df4d0aa5447f952b956788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.684221 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9c89f96a304ec3ba946d9f55fdd603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.684170 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 (DEFAULT_TABLE table=test-table [id=abb00550f6df4d0aa5447f952b956788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.684481 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9c89f96a304ec3ba946d9f55fdd603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.687052 4922 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Bootstrap starting.
I20260927 06:59:07.687352 4923 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Bootstrap starting.
I20260927 06:59:07.687681 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 (DEFAULT_TABLE table=test-table [id=abb00550f6df4d0aa5447f952b956788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.687865 4922 tablet_bootstrap.cc:654] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.687989 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d9c89f96a304ec3ba946d9f55fdd603. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.688012 4923 tablet_bootstrap.cc:654] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.688189 4922 log.cc:824] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:07.688337 4923 log.cc:824] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:07.689020 4922 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: No bootstrap required, opened a new log
I20260927 06:59:07.689071 4923 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: No bootstrap required, opened a new log
I20260927 06:59:07.689092 4922 ts_tablet_manager.cc:1397] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:07.689117 4923 ts_tablet_manager.cc:1397] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:07.690768 4926 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Bootstrap starting.
I20260927 06:59:07.691044 4922 raft_consensus.cc:348] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.691044 4923 raft_consensus.cc:348] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.691210 4922 raft_consensus.cc:374] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.691210 4923 raft_consensus.cc:374] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.691247 4923 raft_consensus.cc:729] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2e59e1dd504615939cf049822899c4, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.691247 4922 raft_consensus.cc:729] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b496ef67084ca8b8e0b219e3d8ada6, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.691365 4922 consensus_queue.cc:260] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.691370 4923 consensus_queue.cc:260] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.691646 4923 ts_tablet_manager.cc:1428] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:07.691646 4922 ts_tablet_manager.cc:1428] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 06:59:07.691781 4619 heartbeater.cc:503] Master 127.3.57.62:45333 was elected leader, sending a full tablet report...
I20260927 06:59:07.691790 4759 heartbeater.cc:503] Master 127.3.57.62:45333 was elected leader, sending a full tablet report...
I20260927 06:59:07.692070 4926 tablet_bootstrap.cc:654] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.692561 4926 log.cc:824] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:07.693348 4926 tablet_bootstrap.cc:492] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: No bootstrap required, opened a new log
I20260927 06:59:07.693423 4926 ts_tablet_manager.cc:1397] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 06:59:07.694890 4926 raft_consensus.cc:348] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.695029 4926 raft_consensus.cc:374] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.695073 4926 raft_consensus.cc:729] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c637cee3af7b46b696872d6232b96cf8, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.695207 4926 consensus_queue.cc:260] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.695454 4926 ts_tablet_manager.cc:1428] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:07.695530 4899 heartbeater.cc:503] Master 127.3.57.62:45333 was elected leader, sending a full tablet report...
W20260927 06:59:07.728924 4760 tablet.cc:2369] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 06:59:07.830062 4620 tablet.cc:2369] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:07.878194 4927 raft_consensus.cc:482] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:07.878320 4927 raft_consensus.cc:504] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.878639 4927 leader_election.cc:288] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:07.881790 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.878854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:42945 (local address 127.3.57.2:38915)
0927 06:59:07.879072 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.879082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.879140 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.879325 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.879325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.879327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:07.879424 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.879425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.880120 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.880121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.881016 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.881018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.881547 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.881619 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.881624 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.881677 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.881717 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":119,"threads_started":1}
I20260927 06:59:07.881963 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.878858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44405 (local address 127.3.57.2:36085)
0927 06:59:07.878922 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:07.878935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:07.878980 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:07.879222 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:07.879223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:07.879226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:07.879313 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.879316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.880109 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.880110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.881041 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:07.881044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.881702 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:07.881786 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.881789 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:07.881847 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 06:59:07.881887 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":76}
I20260927 06:59:07.881994 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.878926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:36085 (local address 127.3.57.1:44405)
0927 06:59:07.878979 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.878981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.878993 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:07.879070 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.879071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.879082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.879152 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:07.879406 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.880007 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881148 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.881611 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881683 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.881692 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.881875 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.881922 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 06:59:07.881836 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.878959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:38915 (local address 127.3.57.3:42945)
0927 06:59:07.879022 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.879024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.879183 (+ 159us) server_negotiation.cc:407] Connection header received
0927 06:59:07.879221 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.879222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.879237 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.879310 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:07.879522 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.880010 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881090 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.881473 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.881565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.881577 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.881700 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.881744 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 06:59:07.882320 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9c89f96a304ec3ba946d9f55fdd603" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2e59e1dd504615939cf049822899c4" is_pre_election: true
I20260927 06:59:07.882483 4572 raft_consensus.cc:2454] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 0.
I20260927 06:59:07.882452 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9c89f96a304ec3ba946d9f55fdd603" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8" is_pre_election: true
I20260927 06:59:07.882566 4853 raft_consensus.cc:2454] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 0.
I20260927 06:59:07.882714 4645 leader_election.cc:302] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2e59e1dd504615939cf049822899c4, 59b496ef67084ca8b8e0b219e3d8ada6; no voters:
I20260927 06:59:07.882905 4927 raft_consensus.cc:2781] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:07.882946 4927 raft_consensus.cc:482] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:07.882962 4927 raft_consensus.cc:3037] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:07.883898 4927 raft_consensus.cc:504] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.884043 4927 leader_election.cc:288] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 election: Requested vote from peers 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:07.884172 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9c89f96a304ec3ba946d9f55fdd603" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2e59e1dd504615939cf049822899c4"
I20260927 06:59:07.884181 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d9c89f96a304ec3ba946d9f55fdd603" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8"
I20260927 06:59:07.884270 4572 raft_consensus.cc:3037] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:07.884265 4853 raft_consensus.cc:3037] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:07.884912 4572 raft_consensus.cc:2454] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 1.
I20260927 06:59:07.885071 4853 raft_consensus.cc:2454] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 1.
I20260927 06:59:07.885156 4645 leader_election.cc:302] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2e59e1dd504615939cf049822899c4, 59b496ef67084ca8b8e0b219e3d8ada6; no voters:
I20260927 06:59:07.885244 4927 raft_consensus.cc:2781] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:07.885401 4927 raft_consensus.cc:686] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 LEADER]: Becoming Leader. State: Replica: 59b496ef67084ca8b8e0b219e3d8ada6, State: Running, Role: LEADER
I20260927 06:59:07.885509 4927 consensus_queue.cc:237] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:07.886344 4415 catalog_manager.cc:5495] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } health_report { overall_health: HEALTHY } } }
W20260927 06:59:07.892037 4900 tablet.cc:2369] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:07.909226 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:07.906744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.3.57.2:36915)
0927 06:59:07.906816 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:07.906817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:07.906826 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:07.906911 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:07.906911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:07.906929 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:07.906998 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:07.907182 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.907703 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.908437 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:07.908616 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:07.908670 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:07.908838 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:07.908929 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:07.908985 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:07.909076 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 06:59:07.909092 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:59:07.911029 4853 raft_consensus.cc:1261] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Refusing update from remote peer 59b496ef67084ca8b8e0b219e3d8ada6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:07.911031 4572 raft_consensus.cc:1261] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Refusing update from remote peer 59b496ef67084ca8b8e0b219e3d8ada6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:07.911363 4932 consensus_queue.cc:1035] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:07.911516 4927 consensus_queue.cc:1035] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:07.914084 4937 mvcc.cc:205] Tried to move back new op lower bound from 7333856657040707584 to 7333856656939606016. Current Snapshot: MvccSnapshot[applied={T|T < 7333856657040707584}]
I20260927 06:59:07.914726 4936 mvcc.cc:205] Tried to move back new op lower bound from 7333856657040707584 to 7333856656939606016. Current Snapshot: MvccSnapshot[applied={T|T < 7333856657040707584 or (T in {7333856657040707584})}]
I20260927 06:59:07.916944 4416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:07.917049 4416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:07.917797 4416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Sending DeleteTablet for 3 replicas of tablet 2d9c89f96a304ec3ba946d9f55fdd603
I20260927 06:59:07.918119 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33212
I20260927 06:59:07.918161 4693 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:50962
I20260927 06:59:07.918314 4833 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d9c89f96a304ec3ba946d9f55fdd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:34474
I20260927 06:59:07.918358 4945 tablet_replica.cc:330] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: stopping tablet replica
I20260927 06:59:07.918448 4945 raft_consensus.cc:2229] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:07.918628 4945 raft_consensus.cc:2258] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:07.918697 4947 tablet_replica.cc:330] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: stopping tablet replica
I20260927 06:59:07.918838 4947 raft_consensus.cc:2229] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:07.918793 4416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:07.918933 4416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:07.919061 4947 raft_consensus.cc:2258] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:07.919339 4945 ts_tablet_manager.cc:1905] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:07.919516 4947 ts_tablet_manager.cc:1905] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:07.920245 4693 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 (DEFAULT_TABLE table=test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.920377 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cf9396c5c7468fa686f6df6cbed9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.921204 4945 ts_tablet_manager.cc:1918] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:07.921267 4945 log.cc:1196] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/2d9c89f96a304ec3ba946d9f55fdd603
I20260927 06:59:07.921348 4947 ts_tablet_manager.cc:1918] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:07.921417 4947 log.cc:1196] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/2d9c89f96a304ec3ba946d9f55fdd603
I20260927 06:59:07.921515 4945 ts_tablet_manager.cc:1939] T 2d9c89f96a304ec3ba946d9f55fdd603 P 2f2e59e1dd504615939cf049822899c4: Deleting consensus metadata
I20260927 06:59:07.921679 4947 ts_tablet_manager.cc:1939] T 2d9c89f96a304ec3ba946d9f55fdd603 P c637cee3af7b46b696872d6232b96cf8: Deleting consensus metadata
I20260927 06:59:07.921893 4401 catalog_manager.cc:4841] TS 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405): tablet 2d9c89f96a304ec3ba946d9f55fdd603 (table test-table [id=abb00550f6df4d0aa5447f952b956788]) successfully deleted
I20260927 06:59:07.922006 4922 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Bootstrap starting.
I20260927 06:59:07.922427 4922 tablet_bootstrap.cc:654] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.922575 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 (DEFAULT_TABLE table=test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.922677 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cf9396c5c7468fa686f6df6cbed9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.923120 4922 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: No bootstrap required, opened a new log
I20260927 06:59:07.923167 4922 ts_tablet_manager.cc:1397] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 06:59:07.923327 4922 raft_consensus.cc:348] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.923401 4922 raft_consensus.cc:374] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.923426 4922 raft_consensus.cc:729] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b496ef67084ca8b8e0b219e3d8ada6, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.923482 4922 consensus_queue.cc:260] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.923605 4922 ts_tablet_manager.cc:1428] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:07.923949 4923 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Bootstrap starting.
I20260927 06:59:07.924338 4923 tablet_bootstrap.cc:654] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.924747 4403 catalog_manager.cc:4841] TS c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945): tablet 2d9c89f96a304ec3ba946d9f55fdd603 (table test-table [id=abb00550f6df4d0aa5447f952b956788]) successfully deleted
I20260927 06:59:07.924885 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 (DEFAULT_TABLE table=test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:07.925005 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cf9396c5c7468fa686f6df6cbed9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:07.925832 4926 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Bootstrap starting.
I20260927 06:59:07.926170 4923 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: No bootstrap required, opened a new log
I20260927 06:59:07.926227 4926 tablet_bootstrap.cc:654] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:07.926227 4923 ts_tablet_manager.cc:1397] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:07.926370 4923 raft_consensus.cc:348] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.926437 4923 raft_consensus.cc:374] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.926465 4923 raft_consensus.cc:729] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2e59e1dd504615939cf049822899c4, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.926517 4923 consensus_queue.cc:260] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.926621 4923 ts_tablet_manager.cc:1428] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:07.926813 4946 tablet_replica.cc:330] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: stopping tablet replica
I20260927 06:59:07.926944 4946 raft_consensus.cc:2229] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:07.926976 4926 tablet_bootstrap.cc:492] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: No bootstrap required, opened a new log
I20260927 06:59:07.927027 4926 ts_tablet_manager.cc:1397] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:07.927103 4946 raft_consensus.cc:2258] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:07.927170 4926 raft_consensus.cc:348] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.927238 4926 raft_consensus.cc:374] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:07.927259 4926 raft_consensus.cc:729] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c637cee3af7b46b696872d6232b96cf8, State: Initialized, Role: FOLLOWER
I20260927 06:59:07.927304 4926 consensus_queue.cc:260] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:07.927407 4926 ts_tablet_manager.cc:1428] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:07.927613 4946 ts_tablet_manager.cc:1905] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:07.929214 4946 ts_tablet_manager.cc:1918] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:07.929306 4946 log.cc:1196] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/2d9c89f96a304ec3ba946d9f55fdd603
I20260927 06:59:07.929553 4946 ts_tablet_manager.cc:1939] T 2d9c89f96a304ec3ba946d9f55fdd603 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting consensus metadata
I20260927 06:59:07.929972 4401 catalog_manager.cc:4841] TS 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2:36915): tablet 2d9c89f96a304ec3ba946d9f55fdd603 (table test-table [id=abb00550f6df4d0aa5447f952b956788]) successfully deleted
I20260927 06:59:08.073469 4928 raft_consensus.cc:482] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:08.073565 4928 raft_consensus.cc:504] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:08.073894 4928 leader_election.cc:288] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2:36915), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:08.076996 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.074188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:44457 (local address 127.3.57.2:36915)
0927 06:59:08.074239 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.074241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.074365 (+ 124us) server_negotiation.cc:407] Connection header received
0927 06:59:08.074408 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.074409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.074420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.074482 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.074668 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.075165 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076252 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.076636 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076699 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.076707 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.076882 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.076929 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:08.076998 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.074096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36915 (local address 127.3.57.1:44457)
0927 06:59:08.074271 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.074283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.074361 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.074501 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.074502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.074504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.074587 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.074590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.075275 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.075277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.076163 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.076166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.076735 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.076803 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.076807 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.076858 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.076888 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260927 06:59:08.077065 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.074251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:47229 (local address 127.3.57.3:42945)
0927 06:59:08.074306 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.074308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.074453 (+ 145us) server_negotiation.cc:407] Connection header received
0927 06:59:08.074502 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.074503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.074512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.074571 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:08.074782 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.075332 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076337 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.076690 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076771 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.076780 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.076952 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.076988 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 06:59:08.077116 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.074145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:42945 (local address 127.3.57.1:47229)
0927 06:59:08.074369 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.074380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.074446 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.074593 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.074593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.074595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:08.074679 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.074681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.075421 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.075422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.076251 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.076254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.076787 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.076788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.076870 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.076875 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.076930 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.076967 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":130,"threads_started":1}
I20260927 06:59:08.077272 4713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf9396c5c7468fa686f6df6cbed9d2" candidate_uuid: "2f2e59e1dd504615939cf049822899c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" is_pre_election: true
I20260927 06:59:08.077339 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf9396c5c7468fa686f6df6cbed9d2" candidate_uuid: "2f2e59e1dd504615939cf049822899c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8" is_pre_election: true
I20260927 06:59:08.077410 4853 raft_consensus.cc:2454] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f2e59e1dd504615939cf049822899c4 in term 0.
I20260927 06:59:08.077418 4713 raft_consensus.cc:2454] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f2e59e1dd504615939cf049822899c4 in term 0.
I20260927 06:59:08.077590 4505 leader_election.cc:302] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2e59e1dd504615939cf049822899c4, 59b496ef67084ca8b8e0b219e3d8ada6; no voters:
I20260927 06:59:08.077754 4928 raft_consensus.cc:2781] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:08.077801 4928 raft_consensus.cc:482] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:08.077821 4928 raft_consensus.cc:3037] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.078478 4928 raft_consensus.cc:504] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:08.078622 4928 leader_election.cc:288] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [CANDIDATE]: Term 1 election: Requested vote from peers 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2:36915), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:08.078752 4713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf9396c5c7468fa686f6df6cbed9d2" candidate_uuid: "2f2e59e1dd504615939cf049822899c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59b496ef67084ca8b8e0b219e3d8ada6"
I20260927 06:59:08.078794 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cf9396c5c7468fa686f6df6cbed9d2" candidate_uuid: "2f2e59e1dd504615939cf049822899c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8"
I20260927 06:59:08.078842 4713 raft_consensus.cc:3037] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.078861 4853 raft_consensus.cc:3037] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.079630 4853 raft_consensus.cc:2454] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f2e59e1dd504615939cf049822899c4 in term 1.
I20260927 06:59:08.079643 4713 raft_consensus.cc:2454] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f2e59e1dd504615939cf049822899c4 in term 1.
I20260927 06:59:08.079818 4505 leader_election.cc:302] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f2e59e1dd504615939cf049822899c4, 59b496ef67084ca8b8e0b219e3d8ada6; no voters:
I20260927 06:59:08.079922 4928 raft_consensus.cc:2781] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:08.080080 4928 raft_consensus.cc:686] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f2e59e1dd504615939cf049822899c4, State: Running, Role: LEADER
I20260927 06:59:08.080137 4928 consensus_queue.cc:237] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } }
I20260927 06:59:08.080904 4416 catalog_manager.cc:5495] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f2e59e1dd504615939cf049822899c4 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "2f2e59e1dd504615939cf049822899c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } health_report { overall_health: UNKNOWN } } }
Sep 27 06:59:08 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:08.131191 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.124327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.3.57.62:45333)
0927 06:59:08.124382 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.124383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.124529 (+ 146us) server_negotiation.cc:407] Connection header received
0927 06:59:08.124577 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.124578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.124586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.124642 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.125057 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.125541 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.126228 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.126399 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.126447 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.128571 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.128573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.128573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.128580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.130003 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.130408 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.130409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.130410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.130465 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.130652 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.130653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.130653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.130764 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.130840 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.130990 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.131075 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 06:59:08.131270 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.124264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45333 (local address 127.0.0.1:33888)
0927 06:59:08.124456 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.124474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.124534 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.124660 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.124661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.124950 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:08.124997 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.124999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.125612 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.125612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.126140 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.126142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.126465 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.126466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.126510 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.127020 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:08.127026 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:08.128473 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:08.130103 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.130111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.130114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.130315 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.130529 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:08.130530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:08.130530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:08.130592 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:08.130829 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:08.130836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:08.130881 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.130969 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.131147 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":77,"thread_start_us":132,"threads_started":1}
I20260927 06:59:08.133409 4416 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 06:59:08.133607 4416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:08.135068 4553 tablet_service.cc:1467] Processing CreateTablet for tablet ec717cf6c362459b822f2916927fb0ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08f10e654d7f40a0a4bca8214b421e3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:08.135057 4833 tablet_service.cc:1467] Processing CreateTablet for tablet ec717cf6c362459b822f2916927fb0ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08f10e654d7f40a0a4bca8214b421e3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:08.135216 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec717cf6c362459b822f2916927fb0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.135222 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec717cf6c362459b822f2916927fb0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.136703 4926 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: Bootstrap starting.
I20260927 06:59:08.136709 4923 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: Bootstrap starting.
I20260927 06:59:08.136801 4693 tablet_service.cc:1467] Processing CreateTablet for tablet ec717cf6c362459b822f2916927fb0ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08f10e654d7f40a0a4bca8214b421e3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:08.136965 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec717cf6c362459b822f2916927fb0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:08.137153 4923 tablet_bootstrap.cc:654] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.137153 4926 tablet_bootstrap.cc:654] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.137840 4926 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: No bootstrap required, opened a new log
I20260927 06:59:08.137889 4926 ts_tablet_manager.cc:1397] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:08.137948 4923 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: No bootstrap required, opened a new log
I20260927 06:59:08.137992 4923 ts_tablet_manager.cc:1397] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:08.138001 4926 raft_consensus.cc:348] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.138048 4926 raft_consensus.cc:374] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.138062 4926 raft_consensus.cc:729] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c637cee3af7b46b696872d6232b96cf8, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.138118 4926 consensus_queue.cc:260] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.138127 4923 raft_consensus.cc:348] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.138203 4923 raft_consensus.cc:374] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.138224 4923 raft_consensus.cc:729] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f2e59e1dd504615939cf049822899c4, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.138267 4926 ts_tablet_manager.cc:1428] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:08.138264 4923 consensus_queue.cc:260] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.138306 4930 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.138350 4923 ts_tablet_manager.cc:1428] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.138370 4930 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.138463 4922 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: Bootstrap starting.
I20260927 06:59:08.138567 4928 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.138661 4928 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.138762 4922 tablet_bootstrap.cc:654] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:08.139467 4922 tablet_bootstrap.cc:492] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: No bootstrap required, opened a new log
I20260927 06:59:08.139518 4922 ts_tablet_manager.cc:1397] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:08.139647 4922 raft_consensus.cc:348] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.139711 4922 raft_consensus.cc:374] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:08.139755 4922 raft_consensus.cc:729] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59b496ef67084ca8b8e0b219e3d8ada6, State: Initialized, Role: FOLLOWER
I20260927 06:59:08.139808 4922 consensus_queue.cc:260] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.139919 4922 ts_tablet_manager.cc:1428] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:08.140004 4932 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.140066 4932 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.218338 4932 raft_consensus.cc:482] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:08.218447 4932 raft_consensus.cc:504] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.218648 4932 leader_election.cc:288] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:08.218928 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec717cf6c362459b822f2916927fb0ea" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8" is_pre_election: true
I20260927 06:59:08.218928 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec717cf6c362459b822f2916927fb0ea" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2e59e1dd504615939cf049822899c4" is_pre_election: true
I20260927 06:59:08.219043 4853 raft_consensus.cc:2454] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 0.
I20260927 06:59:08.219043 4572 raft_consensus.cc:2454] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 0.
I20260927 06:59:08.219260 4647 leader_election.cc:302] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b496ef67084ca8b8e0b219e3d8ada6, c637cee3af7b46b696872d6232b96cf8; no voters:
I20260927 06:59:08.219426 4932 raft_consensus.cc:2781] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:08.219506 4932 raft_consensus.cc:482] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:08.219539 4932 raft_consensus.cc:3037] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.220260 4932 raft_consensus.cc:504] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.220438 4932 leader_election.cc:288] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 election: Requested vote from peers 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405), c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945)
I20260927 06:59:08.220605 4572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec717cf6c362459b822f2916927fb0ea" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f2e59e1dd504615939cf049822899c4"
I20260927 06:59:08.220614 4853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec717cf6c362459b822f2916927fb0ea" candidate_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c637cee3af7b46b696872d6232b96cf8"
I20260927 06:59:08.220714 4853 raft_consensus.cc:3037] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.220715 4572 raft_consensus.cc:3037] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:08.221282 4853 raft_consensus.cc:2454] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 1.
I20260927 06:59:08.221477 4647 leader_election.cc:302] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59b496ef67084ca8b8e0b219e3d8ada6, c637cee3af7b46b696872d6232b96cf8; no voters:
I20260927 06:59:08.221597 4932 raft_consensus.cc:2781] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:08.221637 4572 raft_consensus.cc:2454] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59b496ef67084ca8b8e0b219e3d8ada6 in term 1.
I20260927 06:59:08.221659 4932 raft_consensus.cc:686] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 LEADER]: Becoming Leader. State: Replica: 59b496ef67084ca8b8e0b219e3d8ada6, State: Running, Role: LEADER
I20260927 06:59:08.221719 4932 consensus_queue.cc:237] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } }
I20260927 06:59:08.222046 4927 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: TxnStatusTablet state changed. Reason: New leader 59b496ef67084ca8b8e0b219e3d8ada6. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.222160 4927 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:08.222325 4965 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 06:59:08.222443 4965 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 06:59:08.222399 4416 catalog_manager.cc:5495] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:08.231518 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.228958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.3.57.2:36915)
0927 06:59:08.228995 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.228997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.229028 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:59:08.229113 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.229113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.229122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.229165 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:08.229301 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.229821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.230920 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.231090 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.231132 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.231315 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.231339 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.231375 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.231452 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.231468 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 06:59:08.231614 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.228908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36915 (local address 127.0.0.1:50966)
0927 06:59:08.228987 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.228998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.229045 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.229179 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.229180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.229181 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:08.229232 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.229234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.229899 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.229900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.230839 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.230841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.231155 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.231156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.231223 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.231239 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.231395 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.231400 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.231451 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.231496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 06:59:08.232517 4853 raft_consensus.cc:1261] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Refusing update from remote peer 59b496ef67084ca8b8e0b219e3d8ada6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.232545 4572 raft_consensus.cc:1261] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Refusing update from remote peer 59b496ef67084ca8b8e0b219e3d8ada6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.232736 4927 consensus_queue.cc:1035] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.232837 4932 consensus_queue.cc:1035] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.233369 4928 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: TxnStatusTablet state changed. Reason: New leader 59b496ef67084ca8b8e0b219e3d8ada6. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.233457 4928 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.233747 4930 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: TxnStatusTablet state changed. Reason: New leader 59b496ef67084ca8b8e0b219e3d8ada6. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.233835 4930 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.234294 4940 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.234382 4940 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:08.234431 4940 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.234472 4940 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 59b496ef67084ca8b8e0b219e3d8ada6: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:08.234520 4935 mvcc.cc:205] Tried to move back new op lower bound from 7333856658358902784 to 7333856658316599296. Current Snapshot: MvccSnapshot[applied={T|T < 7333856658358902784 or (T in {7333856658358902784})}]
I20260927 06:59:08.234613 4930 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.234685 4930 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P c637cee3af7b46b696872d6232b96cf8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.235769 4928 tablet_replica.cc:439] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f2e59e1dd504615939cf049822899c4" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44405 } } peers { permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 } } peers { permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 } } }
I20260927 06:59:08.235859 4928 tablet_replica.cc:442] T ec717cf6c362459b822f2916927fb0ea P 2f2e59e1dd504615939cf049822899c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:08.238556 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.235986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.3.57.1:44405)
0927 06:59:08.236023 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.236025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.236034 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:08.236126 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.236127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.236146 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.236210 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:08.236356 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.236869 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.237907 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.238083 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.238134 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.238293 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.238380 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.238426 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.238497 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.238512 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 06:59:08.243428 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.241096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:48745 (local address 127.3.57.2:36915)
0927 06:59:08.241139 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.241141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.241276 (+ 135us) server_negotiation.cc:407] Connection header received
0927 06:59:08.241340 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.241341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.241353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.241404 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:08.241543 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.241968 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.242715 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.242939 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.243012 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.243184 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.243217 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.243288 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.243348 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.243365 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:08.243475 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.241052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36915 (local address 127.3.57.1:48745)
0927 06:59:08.241198 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.241207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.241293 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.241408 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.241408 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.241409 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:08.241473 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.241475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.242045 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.242046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.242615 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.242617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.243012 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.243013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.243078 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.243099 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.243288 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.243290 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.243343 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.243383 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260927 06:59:08.245616 4853 raft_consensus.cc:1261] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Refusing update from remote peer 2f2e59e1dd504615939cf049822899c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.245765 4713 raft_consensus.cc:1261] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Refusing update from remote peer 2f2e59e1dd504615939cf049822899c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:08.245924 4953 consensus_queue.cc:1035] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c637cee3af7b46b696872d6232b96cf8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.246040 4928 consensus_queue.cc:1035] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59b496ef67084ca8b8e0b219e3d8ada6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:08.257944 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.251998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:47861 (local address 127.3.57.1:44405)
0927 06:59:08.252046 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.252048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.255727 (+ 3679us) server_negotiation.cc:407] Connection header received
0927 06:59:08.255788 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.255789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.255800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.255853 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:08.255980 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.256443 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.257206 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.257410 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.257503 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.257678 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.257708 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.257775 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.257835 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.257854 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 06:59:08.257946 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.251928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44405 (local address 127.3.57.2:47861)
0927 06:59:08.255668 (+ 3740us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:08.255680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:08.255740 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:08.255857 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:08.255858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:08.255859 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:08.255901 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.255902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.256534 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.256535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.257099 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:08.257101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.257505 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.257506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:08.257566 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.257579 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:08.257774 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:08.257776 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:08.257830 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 06:59:08.257874 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3685,"thread_start_us":94,"threads_started":1}
I20260927 06:59:08.268085 4416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:08.268184 4416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:08.268793 4416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1729dd50439344b88931e5347597523b: Sending DeleteTablet for 3 replicas of tablet 24cf9396c5c7468fa686f6df6cbed9d2
I20260927 06:59:08.269052 4553 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33212
I20260927 06:59:08.269153 4945 tablet_replica.cc:330] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: stopping tablet replica
I20260927 06:59:08.269135 4833 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:34474
I20260927 06:59:08.269214 4945 raft_consensus.cc:2229] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:08.269249 4947 tablet_replica.cc:330] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: stopping tablet replica
I20260927 06:59:08.269317 4945 raft_consensus.cc:2258] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.269330 4947 raft_consensus.cc:2229] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:08.269333 4693 tablet_service.cc:1514] Processing DeleteTablet for tablet 24cf9396c5c7468fa686f6df6cbed9d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:08 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:50962
I20260927 06:59:08.269369 4947 raft_consensus.cc:2258] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.269426 4946 tablet_replica.cc:330] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: stopping tablet replica
I20260927 06:59:08.269490 4946 raft_consensus.cc:2229] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:08.269524 4946 raft_consensus.cc:2258] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:08.269745 4947 ts_tablet_manager.cc:1905] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:08.270169 4946 ts_tablet_manager.cc:1905] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:08.271461 4945 ts_tablet_manager.cc:1905] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:08.271621 4946 ts_tablet_manager.cc:1918] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:08.271621 4947 ts_tablet_manager.cc:1918] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:08.271683 4947 log.cc:1196] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/24cf9396c5c7468fa686f6df6cbed9d2
I20260927 06:59:08.271683 4946 log.cc:1196] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/24cf9396c5c7468fa686f6df6cbed9d2
I20260927 06:59:08.271912 4946 ts_tablet_manager.cc:1939] T 24cf9396c5c7468fa686f6df6cbed9d2 P 59b496ef67084ca8b8e0b219e3d8ada6: Deleting consensus metadata
I20260927 06:59:08.271914 4947 ts_tablet_manager.cc:1939] T 24cf9396c5c7468fa686f6df6cbed9d2 P c637cee3af7b46b696872d6232b96cf8: Deleting consensus metadata
I20260927 06:59:08.272192 4403 catalog_manager.cc:4841] TS c637cee3af7b46b696872d6232b96cf8 (127.3.57.3:42945): tablet 24cf9396c5c7468fa686f6df6cbed9d2 (table test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]) successfully deleted
I20260927 06:59:08.272192 4401 catalog_manager.cc:4841] TS 59b496ef67084ca8b8e0b219e3d8ada6 (127.3.57.2:36915): tablet 24cf9396c5c7468fa686f6df6cbed9d2 (table test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]) successfully deleted
I20260927 06:59:08.273084 4945 ts_tablet_manager.cc:1918] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:08.273149 4945 log.cc:1196] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/24cf9396c5c7468fa686f6df6cbed9d2
I20260927 06:59:08.273380 4945 ts_tablet_manager.cc:1939] T 24cf9396c5c7468fa686f6df6cbed9d2 P 2f2e59e1dd504615939cf049822899c4: Deleting consensus metadata
I20260927 06:59:08.273675 4401 catalog_manager.cc:4841] TS 2f2e59e1dd504615939cf049822899c4 (127.3.57.1:44405): tablet 24cf9396c5c7468fa686f6df6cbed9d2 (table test-table [id=520519fcf4984ba0ac4ed6b19ad1f10b]) successfully deleted
Sep 27 06:59:08 dist-test-slave-mk3h krb5kdc[4371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:08.283713 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.270062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.3.57.1:44405)
0927 06:59:08.270097 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.270099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.275729 (+ 5630us) server_negotiation.cc:407] Connection header received
0927 06:59:08.275802 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.275803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.275814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.275870 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.276405 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.276861 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.277568 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.277751 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.277821 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.280696 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.280704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.280704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.280711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.282464 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.282839 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.282842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.282842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.282911 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.283122 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.283123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.283124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.283262 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.283405 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.283494 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.283588 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 06:59:08.283954 4595 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33236
I20260927 06:59:08.290596 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:08.284772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.3.57.62:45333)
0927 06:59:08.284823 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:08.284825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:08.284980 (+ 155us) server_negotiation.cc:407] Connection header received
0927 06:59:08.285051 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:08.285052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:08.285062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:08.285117 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:08.285584 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.286071 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.286713 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:08.286914 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:08.287005 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:08.288083 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:08.288084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:08.288085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:08.288091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:08.289510 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.289833 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.289834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.289835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.289885 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:08.290077 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:08.290077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:08.290078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:08.290213 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:08.290317 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:08.290424 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 06:59:08.290501 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 06:59:08.290879 4416 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33898
I20260927 06:59:08.291467 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4488
I20260927 06:59:08.297691 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4628
I20260927 06:59:08.302973 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4768
I20260927 06:59:08.307336 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4384
2026-09-27T06:59:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2429 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:08 dist-test-slave-mk3h krb5kdc[4986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:08 dist-test-slave-mk3h krb5kdc[4986](info): set up 2 sockets
Sep 27 06:59:08 dist-test-slave-mk3h krb5kdc[4986](info): commencing operation
krb5kdc: starting...
W20260927 06:59:09.335228 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:09.455837 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34411
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:34411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:09.538640 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:09.538867 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:09.538890 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:09.540117 4999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:09.540158 4999 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 06:59:09.540179 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:09.540191 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:09.540203 4999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:09.540216 4999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:09.541608 4999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:34411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:09.541796 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:09.541958 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:09.546759 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.546767 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.546767 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:09.546800 4999 server_base.cc:1034] running on GCE node
I20260927 06:59:09.547230 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:09.547609 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:09.548774 4999 hybrid_clock.cc:648] HybridClock initialized: now 1790492349548747 us; error 39 us; skew 500 ppm
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:09.551224 4999 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:09.551769 4999 webserver.cc:460] Webserver started at http://127.3.57.62:40897/ using document root <none> and password file <none>
I20260927 06:59:09.552013 4999 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:09.552080 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:09.552212 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:09.553210 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "4c5b9837a3bc4267846846987e90dfeb"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "b87c9034309d4cba6a622fbfa3626057"
server_key_iv: "6d8b9462ab12b7e3dcd8d4917030d88f"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.553584 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "4c5b9837a3bc4267846846987e90dfeb"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "b87c9034309d4cba6a622fbfa3626057"
server_key_iv: "6d8b9462ab12b7e3dcd8d4917030d88f"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.555124 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 06:59:09.556085 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.556320 4999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 06:59:09.556420 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "4c5b9837a3bc4267846846987e90dfeb"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "b87c9034309d4cba6a622fbfa3626057"
server_key_iv: "6d8b9462ab12b7e3dcd8d4917030d88f"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.556509 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:09.565366 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:09.567233 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:09.567385 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:09.571954 4999 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:34411
I20260927 06:59:09.572001 5076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:34411 every 8 connection(s)
I20260927 06:59:09.572295 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:09.572839 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:09.574996 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Bootstrap starting.
I20260927 06:59:09.575721 5077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:09.576004 5077 log.cc:824] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:09.576649 5077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: No bootstrap required, opened a new log
I20260927 06:59:09.578711 5077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } }
I20260927 06:59:09.578842 5077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:09.578864 5077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b9837a3bc4267846846987e90dfeb, State: Initialized, Role: FOLLOWER
I20260927 06:59:09.579028 5077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } }
I20260927 06:59:09.579088 5077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:09.579133 5077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:09.579191 5077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:09.579756 5077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } }
I20260927 06:59:09.579864 5077 leader_election.cc:302] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5b9837a3bc4267846846987e90dfeb; no voters:
I20260927 06:59:09.580045 5077 leader_election.cc:288] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:09.580108 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:09.580261 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [term 1 LEADER]: Becoming Leader. State: Replica: 4c5b9837a3bc4267846846987e90dfeb, State: Running, Role: LEADER
I20260927 06:59:09.580281 5077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:09.580374 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } }
I20260927 06:59:09.580767 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5b9837a3bc4267846846987e90dfeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } } }
I20260927 06:59:09.580786 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5b9837a3bc4267846846987e90dfeb. Latest consensus state: current_term: 1 leader_uuid: "4c5b9837a3bc4267846846987e90dfeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b9837a3bc4267846846987e90dfeb" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34411 } } }
I20260927 06:59:09.580906 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:09.580942 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:09.581334 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 4999
I20260927 06:59:09.581432 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:09.581630 3300 external_mini_cluster.cc:1428] Setting key 9256ba1e1ab766904048059589484a7d
I20260927 06:59:09.581988 5087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:09.582518 5087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:09.584043 5087 catalog_manager.cc:1347] Generated new cluster ID: 736a1d1fe356473282b6c315ab334e11
I20260927 06:59:09.584096 5087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:09.591158 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.582101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.3.57.62:34411)
0927 06:59:09.582243 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.582253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.582313 (+ 60us) server_negotiation.cc:407] Connection header received
0927 06:59:09.582448 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.582455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.582506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.582685 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.583361 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.584524 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.585284 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.585680 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.585799 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.587994 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.587998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.587998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.588010 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.589617 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.589971 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.589974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.589974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.590065 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.590301 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.590301 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.590302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.590524 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.590678 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.590796 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.591008 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260927 06:59:09.596958 5087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:09.597481 5087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:09.604144 5087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Generated new TSK 0
I20260927 06:59:09.604315 5087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:09.637856 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34411
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:09.723064 5103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:09.723264 5103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:09.723290 5103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:09.724506 5103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:09.724548 5103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:09.724570 5103 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:59:09.724619 5103 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:09.724646 5103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:09.726128 5103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:09.726361 5103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:09.726610 5103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:09.731513 5111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.731521 5108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:09.731658 5103 server_base.cc:1034] running on GCE node
W20260927 06:59:09.731527 5109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:09.731988 5103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:09.732182 5103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:09.733317 5103 hybrid_clock.cc:648] HybridClock initialized: now 1790492349733304 us; error 30 us; skew 500 ppm
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:09.736548 5103 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:09.737084 5103 webserver.cc:460] Webserver started at http://127.3.57.1:39109/ using document root <none> and password file <none>
I20260927 06:59:09.737352 5103 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:09.737421 5103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:09.737566 5103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:09.738997 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "eb0f453aa3ea466591ad363641fc79d0"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "12bb61e369d1f1a11a4f0961f9acb5dd"
server_key_iv: "a46af13880f0dd57a7a0e2f06afbde30"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.739466 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0f453aa3ea466591ad363641fc79d0"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "12bb61e369d1f1a11a4f0961f9acb5dd"
server_key_iv: "a46af13880f0dd57a7a0e2f06afbde30"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.741362 5103 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:09.742345 5118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.742587 5103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:09.742682 5103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "eb0f453aa3ea466591ad363641fc79d0"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "12bb61e369d1f1a11a4f0961f9acb5dd"
server_key_iv: "a46af13880f0dd57a7a0e2f06afbde30"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.742758 5103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:09.780812 5103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:09.783221 5103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:09.783370 5103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:09.783918 5103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:09.783954 5103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.783981 5103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:09.784018 5103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.790957 5103 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:32855
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:09.791046 5233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:32855 every 8 connection(s)
I20260927 06:59:09.791324 5103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:09.792817 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5103
I20260927 06:59:09.792936 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:09.793107 3300 external_mini_cluster.cc:1428] Setting key 38914bc943fbdb8b3065234bd3869ff7
I20260927 06:59:09.794245 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.784495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:35681 (local address 127.3.57.62:34411)
0927 06:59:09.784553 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.784556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.784803 (+ 247us) server_negotiation.cc:407] Connection header received
0927 06:59:09.785404 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.785406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.785426 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.785498 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.786455 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.787311 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.788437 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.788827 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.788926 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.791636 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.791637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.791638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.791645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.792820 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.793275 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.793276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.793277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.793342 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.793601 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.793602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.793602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.793719 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.793795 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.794063 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.794147 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 06:59:09.794432 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.784478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.1:35681)
0927 06:59:09.784674 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:09.784712 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:09.785251 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:09.785608 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:09.785614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:09.786042 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:09.786330 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.786337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.787419 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.787420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.788365 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.788368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.788960 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.788961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.789090 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.789654 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:09.789665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:09.791517 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:09.792909 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.792916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.792918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.793212 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.793412 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.793413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.793414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.793526 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.793786 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:09.793793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:09.793930 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 06:59:09.794038 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 06:59:09.794202 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260927 06:59:09.795071 5234 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34411
I20260927 06:59:09.795169 5234 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:09.795363 5234 heartbeater.cc:511] Master 127.3.57.62:34411 requested a full tablet report, sending...
I20260927 06:59:09.795917 5030 ts_manager.cc:194] Registered new tserver with Master: eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855)
I20260927 06:59:09.796627 5030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:35681
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
I20260927 06:59:09.805323 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.796159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:39737 (local address 127.3.57.62:34411)
0927 06:59:09.796220 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.796222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.799753 (+ 3531us) server_negotiation.cc:407] Connection header received
0927 06:59:09.799827 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.799828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.799838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.799893 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.800231 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.800747 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.801578 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.801783 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.801858 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.802712 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.802713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.802713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.802720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.804190 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.804546 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.804547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.804548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.804602 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.804812 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.804813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.804814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.804927 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.805002 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.805150 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.805230 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 06:59:09.805335 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.796008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.1:39737)
0927 06:59:09.799678 (+ 3670us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:09.799688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:09.799766 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:09.799900 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:09.799900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:09.800073 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:09.800136 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.800137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.800845 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.800846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.801480 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.801483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.801855 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.801856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.801929 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.802295 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:09.802299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:09.802591 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:09.804295 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.804296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.804297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.804471 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.804679 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.804680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.804681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.804738 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.804995 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:09.804996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:09.805038 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 06:59:09.805126 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 06:59:09.805229 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3626,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:09.818459 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34411
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:09.899472 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:09.899705 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:09.899731 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:09.900882 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:09.900920 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:09.900941 5243 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:59:09.900954 5243 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:09.900970 5243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:09.902488 5243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:09.902669 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:09.902841 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:09.908165 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.908205 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:09.908181 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:09.908433 5243 server_base.cc:1034] running on GCE node
I20260927 06:59:09.908699 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:09.908900 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:09.910041 5243 hybrid_clock.cc:648] HybridClock initialized: now 1790492349910027 us; error 28 us; skew 500 ppm
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:09.912580 5243 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:09.912988 5243 webserver.cc:460] Webserver started at http://127.3.57.2:32987/ using document root <none> and password file <none>
I20260927 06:59:09.913189 5243 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:09.913230 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:09.913317 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:09.914297 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "af6dacf92b63475fa725d864a7a708d8"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "33e6d971d3a50bb32ca4fbb970ed5348"
server_key_iv: "005f308e1da1a1a4cacdd433657e8f17"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.914588 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "af6dacf92b63475fa725d864a7a708d8"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "33e6d971d3a50bb32ca4fbb970ed5348"
server_key_iv: "005f308e1da1a1a4cacdd433657e8f17"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.915894 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:09.916580 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.916759 5243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:09.916862 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "af6dacf92b63475fa725d864a7a708d8"
format_stamp: "Formatted at 2026-09-27 06:59:09 on dist-test-slave-mk3h"
server_key: "33e6d971d3a50bb32ca4fbb970ed5348"
server_key_iv: "005f308e1da1a1a4cacdd433657e8f17"
server_key_version: "encryptionkey@0"
I20260927 06:59:09.916942 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:09.927284 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:09.929553 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:09.929713 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:09.930217 5243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:09.930282 5243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.930327 5243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:09.930351 5243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:09.937712 5243 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46569
I20260927 06:59:09.938011 5373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46569 every 8 connection(s)
I20260927 06:59:09.938120 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 06:59:09 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:09.942068 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.930736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:36083 (local address 127.3.57.62:34411)
0927 06:59:09.930807 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.930809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.931047 (+ 238us) server_negotiation.cc:407] Connection header received
0927 06:59:09.931832 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.931834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.931848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.931916 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.932832 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.933723 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.935369 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.935749 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.935828 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.938982 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.938983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.938984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.938990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.940443 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.941116 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.941116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.941117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.941168 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.941450 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.941450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.941451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.941554 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.941605 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.941940 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.942005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 06:59:09.942314 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.930700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.2:36083)
0927 06:59:09.930908 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:09.930955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:09.931671 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:09.932023 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:09.932028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:09.932407 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:09.932738 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.932745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.933815 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.933816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.935272 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.935275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.935831 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.935832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.935990 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.936811 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:09.936821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:09.938855 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:09.940699 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.940705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.940707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.941036 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.941234 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.941235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.941235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.941386 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.941616 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:09.941621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:09.941796 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 06:59:09.941909 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 06:59:09.942048 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260927 06:59:09.942847 5374 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34411
I20260927 06:59:09.942929 5374 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:09.943111 5374 heartbeater.cc:511] Master 127.3.57.62:34411 requested a full tablet report, sending...
I20260927 06:59:09.943521 5030 ts_manager.cc:194] Registered new tserver with Master: af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:09.944051 5030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:36083
I20260927 06:59:09.944077 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5243
I20260927 06:59:09.944257 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:09.944464 3300 external_mini_cluster.cc:1428] Setting key 19ccf35bf98f2199068ed1935ac77962
I20260927 06:59:09.950289 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.943863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.2:52837)
0927 06:59:09.944042 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:09.944060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:09.944132 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:09.944292 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:09.944293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:09.944489 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:09.944590 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.944592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.945259 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.945260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.946189 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:09.946192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.946547 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.946548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:09.946632 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.947084 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:09.947088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:09.947411 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:09.949255 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.949256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.949256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.949436 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.949637 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:09.949637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:09.949638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:09.949678 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:09.950018 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:09.950018 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:09.950052 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:59:09.950102 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 06:59:09.950202 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260927 06:59:09.950639 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:09.943947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:52837 (local address 127.3.57.62:34411)
0927 06:59:09.943989 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:09.943992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:09.944122 (+ 130us) server_negotiation.cc:407] Connection header received
0927 06:59:09.944213 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:09.944213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:09.944231 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:09.944287 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:09.944667 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.945158 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.946274 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:09.946451 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:09.946533 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:09.947506 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:09.947508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:09.947508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:09.947517 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:09.949153 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.949508 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.949509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.949509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.949570 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:09.949748 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:09.949749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:09.949749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:09.949946 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:09.950024 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:09.950137 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 06:59:09.950251 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:09.974361 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34411
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:10.062150 5383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:10.062382 5383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:10.062408 5383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:10.063712 5383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:10.063762 5383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:10.063783 5383 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 06:59:10.063802 5383 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 06:59:10.063820 5383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:10.065392 5383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:34411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:10.065658 5383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:10.065903 5383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:10.071028 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.071015 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:10.071094 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:10.071128 5383 server_base.cc:1034] running on GCE node
I20260927 06:59:10.071409 5383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:10.071682 5383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:10.072880 5383 hybrid_clock.cc:648] HybridClock initialized: now 1790492350072880 us; error 38 us; skew 500 ppm
Sep 27 06:59:10 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:10.075519 5383 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:10.075975 5383 webserver.cc:460] Webserver started at http://127.3.57.3:36679/ using document root <none> and password file <none>
I20260927 06:59:10.076160 5383 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:10.076205 5383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:10.076288 5383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:10.077330 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "b026a20da9724c08ac646153e4a0385b"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-mk3h"
server_key: "619382861c3246036e18b5956c3acf11"
server_key_iv: "13e2739071e836b2b6d9e948b4133e3e"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.077625 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "b026a20da9724c08ac646153e4a0385b"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-mk3h"
server_key: "619382861c3246036e18b5956c3acf11"
server_key_iv: "13e2739071e836b2b6d9e948b4133e3e"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.078860 5383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:10.079640 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.079818 5383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:10.079916 5383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "b026a20da9724c08ac646153e4a0385b"
format_stamp: "Formatted at 2026-09-27 06:59:10 on dist-test-slave-mk3h"
server_key: "619382861c3246036e18b5956c3acf11"
server_key_iv: "13e2739071e836b2b6d9e948b4133e3e"
server_key_version: "encryptionkey@0"
I20260927 06:59:10.079998 5383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:10.107250 5383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:10.109117 5383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:10.109229 5383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:10.109755 5383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:10.109802 5383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.109825 5383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:10.109838 5383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.116430 5383 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33321
I20260927 06:59:10.116500 5513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33321 every 8 connection(s)
I20260927 06:59:10.116776 5383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
Sep 27 06:59:10 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492350, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:10.119493 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5383
I20260927 06:59:10.119654 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:10.119874 3300 external_mini_cluster.cc:1428] Setting key 4bb9a8ac36186c2944329fbf4610e53b
I20260927 06:59:10.119876 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.110292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:54675 (local address 127.3.57.62:34411)
0927 06:59:10.110351 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.110353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.110565 (+ 212us) server_negotiation.cc:407] Connection header received
0927 06:59:10.111103 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.111105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.111119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.111176 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.112020 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.112867 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.114381 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.114717 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.114811 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.117342 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.117344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.117344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:10.117350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.118533 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.118887 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.118888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.118888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.118939 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.119158 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.119158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.119159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.119253 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.119300 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.119691 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.119782 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 06:59:10.120083 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.110269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.3:54675)
0927 06:59:10.110448 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.110483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.110999 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.111246 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.111251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.111654 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:10.111933 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.111940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.112958 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.112959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.114285 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.114288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.114813 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.114814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.114911 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.115503 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:10.115514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:10.117247 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:10.118607 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.118613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.118614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.118823 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.119011 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.119012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.119012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.119108 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.119305 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:10.119309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:10.119416 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.119663 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.119838 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260927 06:59:10.120587 5514 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34411
I20260927 06:59:10.120736 5514 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:10.120925 5514 heartbeater.cc:511] Master 127.3.57.62:34411 requested a full tablet report, sending...
I20260927 06:59:10.121292 5030 ts_manager.cc:194] Registered new tserver with Master: b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321)
I20260927 06:59:10.121832 5030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:54675
I20260927 06:59:10.127949 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.122724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:49263 (local address 127.3.57.62:34411)
0927 06:59:10.122797 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.122800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.122872 (+ 72us) server_negotiation.cc:407] Connection header received
0927 06:59:10.122944 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.122945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.122956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.123011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.123330 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.123864 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.124551 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.124738 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.124846 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.125707 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.125708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.125709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:10.125715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.126828 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.127180 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.127181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.127181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.127238 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.127441 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.127442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.127442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.127581 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.127644 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.127793 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.127861 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:59:10.128194 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.122645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.3.57.3:49263)
0927 06:59:10.122818 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.122829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.122889 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.123015 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.123016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.123185 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:10.123234 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.123235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.123959 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.123960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.124453 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.124455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.124846 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.124846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.124925 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.125330 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:10.125334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:10.125619 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:10.126922 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.126923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.126924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.127095 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.127306 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.127307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.127307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.127369 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.127636 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:10.127637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:10.127679 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.127768 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.127862 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260927 06:59:10.131635 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:59:10 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:59:10 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:10.153403 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.146641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45810 (local address 127.3.57.62:34411)
0927 06:59:10.146700 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.146702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.146713 (+ 11us) server_negotiation.cc:407] Connection header received
0927 06:59:10.146782 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.146783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.146793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.146843 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.147363 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.147891 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.148578 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.148759 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.148813 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.151007 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.151009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.151009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:10.151015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.152239 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.152545 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.152546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.152546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.152605 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.152828 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.152829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.152829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.152923 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.152996 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.153178 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.153245 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 06:59:10.155066 5030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:10.156826 5030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:10.163973 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.159932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.3.57.1:32855)
0927 06:59:10.160081 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.160086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.160384 (+ 298us) server_negotiation.cc:407] Connection header received
0927 06:59:10.160426 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.160428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.160442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.160508 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.160665 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.161333 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.162758 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.163503 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.163628 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.163697 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.163823 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.163869 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260927 06:59:10.163995 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.159656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33321 (local address 127.0.0.1:60874)
0927 06:59:10.159767 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.159806 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.159904 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.160385 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.160388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.160390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.160463 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.160465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.161762 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.161763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.162812 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.162814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.163688 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.163689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.163796 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.163800 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.163848 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.163893 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260927 06:59:10.164238 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.159926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.3.57.3:33321)
0927 06:59:10.160189 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.160193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.160203 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:10.160243 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.160244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.160262 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.160374 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.160548 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.161671 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.162882 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.163599 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.164036 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.164106 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.164130 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.164170 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260927 06:59:10.163973 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.159680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:32855 (local address 127.0.0.1:57014)
0927 06:59:10.160274 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.160285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.160326 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.160570 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.160571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.160572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.160604 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.160605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.161426 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.161427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.162672 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.162675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.163631 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.163633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.163712 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.163726 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.163797 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.163850 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260927 06:59:10.164888 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.159680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.0.0.1:46938)
0927 06:59:10.160132 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.160143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.160191 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.160481 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.160482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.160484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.160529 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.160530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.162878 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.162879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.163530 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.163532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.164645 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.164646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.164736 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.164739 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.164789 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.164826 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":92,"threads_started":1}
I20260927 06:59:10.165136 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.160128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.3.57.2:46569)
0927 06:59:10.160316 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.160320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.160342 (+ 22us) server_negotiation.cc:407] Connection header received
0927 06:59:10.160385 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.160387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.160402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.160486 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.160589 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.161672 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.163697 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.164473 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.164562 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.164617 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.164995 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.165037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260927 06:59:10.165132 5168 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3a52e178543678918c6cc03c505c6 (DEFAULT_TABLE table=test-table [id=2876503384aa43f29f5de250f58d3d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.165258 5448 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3a52e178543678918c6cc03c505c6 (DEFAULT_TABLE table=test-table [id=2876503384aa43f29f5de250f58d3d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.165475 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3a52e178543678918c6cc03c505c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.165593 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3a52e178543678918c6cc03c505c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.165721 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 06b3a52e178543678918c6cc03c505c6 (DEFAULT_TABLE table=test-table [id=2876503384aa43f29f5de250f58d3d46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.166010 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06b3a52e178543678918c6cc03c505c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.168280 5537 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Bootstrap starting.
I20260927 06:59:10.168780 5537 tablet_bootstrap.cc:654] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.169003 5537 log.cc:824] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:10.169787 5540 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Bootstrap starting.
I20260927 06:59:10.169839 5537 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: No bootstrap required, opened a new log
I20260927 06:59:10.169919 5537 ts_tablet_manager.cc:1397] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.169909 5539 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Bootstrap starting.
I20260927 06:59:10.170555 5539 tablet_bootstrap.cc:654] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.170555 5540 tablet_bootstrap.cc:654] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.170915 5539 log.cc:824] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:10.170966 5540 log.cc:824] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:10.171727 5539 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: No bootstrap required, opened a new log
I20260927 06:59:10.171808 5539 ts_tablet_manager.cc:1397] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.171819 5540 tablet_bootstrap.cc:492] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: No bootstrap required, opened a new log
I20260927 06:59:10.171870 5540 ts_tablet_manager.cc:1397] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.171770 5537 raft_consensus.cc:348] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.171938 5537 raft_consensus.cc:374] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.171981 5537 raft_consensus.cc:729] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.172103 5537 consensus_queue.cc:260] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.172381 5537 ts_tablet_manager.cc:1428] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:10.172422 5234 heartbeater.cc:503] Master 127.3.57.62:34411 was elected leader, sending a full tablet report...
I20260927 06:59:10.173359 5540 raft_consensus.cc:348] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.173507 5540 raft_consensus.cc:374] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.173550 5540 raft_consensus.cc:729] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.173674 5540 consensus_queue.cc:260] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.173839 5539 raft_consensus.cc:348] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.173964 5540 ts_tablet_manager.cc:1428] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:10.174000 5374 heartbeater.cc:503] Master 127.3.57.62:34411 was elected leader, sending a full tablet report...
I20260927 06:59:10.174005 5539 raft_consensus.cc:374] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.174048 5539 raft_consensus.cc:729] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.174155 5539 consensus_queue.cc:260] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.174417 5539 ts_tablet_manager.cc:1428] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 06:59:10.174480 5514 heartbeater.cc:503] Master 127.3.57.62:34411 was elected leader, sending a full tablet report...
W20260927 06:59:10.188865 5375 tablet.cc:2369] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:10.280741 5543 raft_consensus.cc:482] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:10.280905 5543 raft_consensus.cc:504] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.281316 5543 leader_election.cc:288] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:10.284368 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.281538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:59429 (local address 127.3.57.3:33321)
0927 06:59:10.281588 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.281590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.281737 (+ 147us) server_negotiation.cc:407] Connection header received
0927 06:59:10.281789 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.281790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.281802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.281856 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.282029 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.282569 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.283604 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.284036 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.284089 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.284098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.284246 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.284285 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 06:59:10.284376 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.281489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33321 (local address 127.3.57.1:59429)
0927 06:59:10.281652 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.281663 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.281729 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.281856 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.281856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.281858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.281955 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.281956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.282667 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.282669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.283466 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.283468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.284094 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.284095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.284174 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.284179 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.284231 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.284269 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":52,"thread_start_us":79,"threads_started":1}
I20260927 06:59:10.284466 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.281623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:38027 (local address 127.3.57.2:46569)
0927 06:59:10.281669 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.281671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.281684 (+ 13us) server_negotiation.cc:407] Connection header received
0927 06:59:10.281731 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.281732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.281744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.281799 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.281961 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.282565 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.283645 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.284088 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.284157 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.284169 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.284346 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.284393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:10.284504 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.281488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.3.57.1:38027)
0927 06:59:10.281613 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.281623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.281662 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.281800 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.281801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.281802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.281889 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.281891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.282668 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.282669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.283493 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.283496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.284174 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.284175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.284266 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.284270 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.284326 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.284364 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":34}
I20260927 06:59:10.285187 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3a52e178543678918c6cc03c505c6" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8" is_pre_election: true
I20260927 06:59:10.285187 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3a52e178543678918c6cc03c505c6" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b" is_pre_election: true
I20260927 06:59:10.285341 5468 raft_consensus.cc:2454] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 0.
I20260927 06:59:10.285341 5326 raft_consensus.cc:2454] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 0.
I20260927 06:59:10.285566 5120 leader_election.cc:302] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6dacf92b63475fa725d864a7a708d8, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:10.285714 5543 raft_consensus.cc:2781] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:10.285771 5543 raft_consensus.cc:482] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:10.285799 5543 raft_consensus.cc:3037] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.286468 5543 raft_consensus.cc:504] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.286605 5543 leader_election.cc:288] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 election: Requested vote from peers b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:10.286748 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3a52e178543678918c6cc03c505c6" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8"
I20260927 06:59:10.286734 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06b3a52e178543678918c6cc03c505c6" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b"
I20260927 06:59:10.286815 5468 raft_consensus.cc:3037] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.286815 5326 raft_consensus.cc:3037] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.287715 5468 raft_consensus.cc:2454] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 1.
I20260927 06:59:10.287715 5326 raft_consensus.cc:2454] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 1.
I20260927 06:59:10.287881 5121 leader_election.cc:302] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b026a20da9724c08ac646153e4a0385b, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:10.287989 5543 raft_consensus.cc:2781] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:10.288148 5543 raft_consensus.cc:686] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 1 LEADER]: Becoming Leader. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Running, Role: LEADER
I20260927 06:59:10.288259 5543 consensus_queue.cc:237] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } }
I20260927 06:59:10.289147 5030 catalog_manager.cc:5495] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "eb0f453aa3ea466591ad363641fc79d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } health_report { overall_health: HEALTHY } } }
W20260927 06:59:10.292402 5235 tablet.cc:2369] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 06:59:10 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:10.335016 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.328211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.3.57.62:34411)
0927 06:59:10.328302 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.328305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.328435 (+ 130us) server_negotiation.cc:407] Connection header received
0927 06:59:10.328518 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.328519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.328529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.328598 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:10.329049 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.329584 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.330287 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.330484 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.330567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.332552 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:10.332553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:10.332553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:10.332561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:10.333789 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.334202 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.334203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.334203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.334265 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:10.334472 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:10.334473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:10.334473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:10.334581 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:10.334657 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.334815 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.334900 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 06:59:10.335094 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.328136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34411 (local address 127.0.0.1:45826)
0927 06:59:10.328354 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.328364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.328448 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.328601 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.328602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.328911 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:10.328955 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.328958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.329665 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.329666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.330198 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.330200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.330568 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.330569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.330629 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.331101 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:10.331105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:10.332457 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:10.333909 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.333916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.333918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.334112 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.334340 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:10.334341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:10.334341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:10.334409 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:10.334648 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:10.334655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:10.334700 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.334792 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.334971 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":80,"thread_start_us":112,"threads_started":1}
I20260927 06:59:10.337069 5030 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:45826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 06:59:10.337265 5030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:10.338727 5448 tablet_service.cc:1467] Processing CreateTablet for tablet f053b6544ee54b7f83b5f3848397b9d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acb6d91a2388469fabae14433638e5f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:10.338726 5168 tablet_service.cc:1467] Processing CreateTablet for tablet f053b6544ee54b7f83b5f3848397b9d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acb6d91a2388469fabae14433638e5f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:10.338729 5308 tablet_service.cc:1467] Processing CreateTablet for tablet f053b6544ee54b7f83b5f3848397b9d7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=acb6d91a2388469fabae14433638e5f6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 06:59:10.338900 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f053b6544ee54b7f83b5f3848397b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.338900 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f053b6544ee54b7f83b5f3848397b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.338901 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f053b6544ee54b7f83b5f3848397b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.340278 5539 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: Bootstrap starting.
I20260927 06:59:10.340444 5537 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: Bootstrap starting.
I20260927 06:59:10.340787 5537 tablet_bootstrap.cc:654] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.340921 5539 tablet_bootstrap.cc:654] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.341085 5540 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: Bootstrap starting.
I20260927 06:59:10.341665 5540 tablet_bootstrap.cc:654] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.342293 5537 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: No bootstrap required, opened a new log
I20260927 06:59:10.342345 5537 ts_tablet_manager.cc:1397] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.342396 5540 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: No bootstrap required, opened a new log
I20260927 06:59:10.342448 5540 ts_tablet_manager.cc:1397] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:10.342499 5537 raft_consensus.cc:348] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.342581 5537 raft_consensus.cc:374] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.342607 5537 raft_consensus.cc:729] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.342588 5540 raft_consensus.cc:348] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.342670 5540 raft_consensus.cc:374] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.342661 5537 consensus_queue.cc:260] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.342701 5540 raft_consensus.cc:729] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.342760 5537 ts_tablet_manager.cc:1428] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:10.342739 5540 consensus_queue.cc:260] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.342832 5540 ts_tablet_manager.cc:1428] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.342902 5543 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.342974 5543 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.343065 5544 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.343158 5544 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.343878 5539 tablet_bootstrap.cc:492] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: No bootstrap required, opened a new log
I20260927 06:59:10.343945 5539 ts_tablet_manager.cc:1397] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 06:59:10.344096 5539 raft_consensus.cc:348] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.344197 5539 raft_consensus.cc:374] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.344219 5539 raft_consensus.cc:729] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.344269 5539 consensus_queue.cc:260] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.344367 5539 ts_tablet_manager.cc:1428] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:10.344434 5545 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.344534 5545 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.362622 5544 raft_consensus.cc:482] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:10.362728 5544 raft_consensus.cc:504] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.363114 5544 leader_election.cc:288] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321)
I20260927 06:59:10.366021 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.363425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:33775 (local address 127.3.57.3:33321)
0927 06:59:10.363466 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.363468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.363630 (+ 162us) server_negotiation.cc:407] Connection header received
0927 06:59:10.363694 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.363695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.363704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.363758 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.363921 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.364386 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365285 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.365637 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365715 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.365726 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.365912 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.365948 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 06:59:10.366057 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.363315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:53069 (local address 127.3.57.1:32855)
0927 06:59:10.363394 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.363397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.363411 (+ 14us) server_negotiation.cc:407] Connection header received
0927 06:59:10.363457 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.363458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.363476 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.363532 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:10.363745 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.364276 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365278 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.365683 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365772 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.365782 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.365939 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.365980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 06:59:10.366076 5558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.363357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33321 (local address 127.3.57.2:33775)
0927 06:59:10.363581 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.363590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.363643 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.363764 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.363765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.363767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.363853 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.363854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.364476 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.364477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.365195 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.365196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.365745 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.365826 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.365829 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.365903 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.365944 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":124,"threads_started":1}
I20260927 06:59:10.366079 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.363265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:32855 (local address 127.3.57.2:53069)
0927 06:59:10.363336 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.363348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.363389 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.363572 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.363574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.363576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:10.363650 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.363651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.364369 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.364370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.365179 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.365181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.365771 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.365772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.365856 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.365859 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.365933 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.365977 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 06:59:10.366348 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f053b6544ee54b7f83b5f3848397b9d7" candidate_uuid: "af6dacf92b63475fa725d864a7a708d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b" is_pre_election: true
I20260927 06:59:10.366451 5468 raft_consensus.cc:2454] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6dacf92b63475fa725d864a7a708d8 in term 0.
I20260927 06:59:10.366641 5261 leader_election.cc:302] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6dacf92b63475fa725d864a7a708d8, b026a20da9724c08ac646153e4a0385b; no voters:
I20260927 06:59:10.366605 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f053b6544ee54b7f83b5f3848397b9d7" candidate_uuid: "af6dacf92b63475fa725d864a7a708d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0" is_pre_election: true
I20260927 06:59:10.366746 5188 raft_consensus.cc:2454] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6dacf92b63475fa725d864a7a708d8 in term 0.
I20260927 06:59:10.366765 5544 raft_consensus.cc:2781] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:10.366799 5544 raft_consensus.cc:482] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:10.366838 5544 raft_consensus.cc:3037] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Advancing to term 1
W20260927 06:59:10.367321 5515 tablet.cc:2369] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:10.367767 5544 raft_consensus.cc:504] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.367910 5544 leader_election.cc:288] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [CANDIDATE]: Term 1 election: Requested vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321)
I20260927 06:59:10.368064 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f053b6544ee54b7f83b5f3848397b9d7" candidate_uuid: "af6dacf92b63475fa725d864a7a708d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b"
I20260927 06:59:10.368096 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f053b6544ee54b7f83b5f3848397b9d7" candidate_uuid: "af6dacf92b63475fa725d864a7a708d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0"
I20260927 06:59:10.368155 5468 raft_consensus.cc:3037] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.368194 5188 raft_consensus.cc:3037] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.368719 5468 raft_consensus.cc:2454] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6dacf92b63475fa725d864a7a708d8 in term 1.
I20260927 06:59:10.368875 5261 leader_election.cc:302] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6dacf92b63475fa725d864a7a708d8, b026a20da9724c08ac646153e4a0385b; no voters:
I20260927 06:59:10.368969 5188 raft_consensus.cc:2454] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6dacf92b63475fa725d864a7a708d8 in term 1.
I20260927 06:59:10.368976 5544 raft_consensus.cc:2781] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:10.369153 5544 raft_consensus.cc:686] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [term 1 LEADER]: Becoming Leader. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Running, Role: LEADER
I20260927 06:59:10.369249 5544 consensus_queue.cc:237] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.369678 5559 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: TxnStatusTablet state changed. Reason: New leader af6dacf92b63475fa725d864a7a708d8. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.369774 5559 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:10.369935 5561 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 06:59:10.370056 5561 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 06:59:10.370039 5031 catalog_manager.cc:5495] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to af6dacf92b63475fa725d864a7a708d8 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } health_report { overall_health: UNKNOWN } } }
I20260927 06:59:10.379611 5554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.377113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.0.0.1:46950)
0927 06:59:10.377168 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.377186 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.377245 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.377420 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.377421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.377422 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:10.377478 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.377480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.378096 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.378097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.378658 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.378660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.379034 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.379035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.379106 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.379120 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.379407 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.379416 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.379466 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.379515 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 06:59:10.379593 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.377189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.3.57.2:46569)
0927 06:59:10.377252 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.377254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.377263 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:10.377334 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.377335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.377352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.377416 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:10.377546 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.378013 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.378744 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.378958 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.379034 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.379223 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.379336 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.379429 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.379472 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.379495 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 06:59:10.381536 5468 raft_consensus.cc:1261] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Refusing update from remote peer af6dacf92b63475fa725d864a7a708d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.381609 5188 raft_consensus.cc:1261] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Refusing update from remote peer af6dacf92b63475fa725d864a7a708d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.381884 5544 consensus_queue.cc:1035] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:10.382031 5559 consensus_queue.cc:1035] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:10.383141 5545 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: TxnStatusTablet state changed. Reason: New leader af6dacf92b63475fa725d864a7a708d8. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.383109 5543 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: TxnStatusTablet state changed. Reason: New leader af6dacf92b63475fa725d864a7a708d8. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.383213 5545 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.383214 5543 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.384559 5559 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.384626 5559 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:10.384785 5564 mvcc.cc:205] Tried to move back new op lower bound from 7333856667160018944 to 7333856667112890368. Current Snapshot: MvccSnapshot[applied={T|T < 7333856667160018944}]
I20260927 06:59:10.384785 5545 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.384873 5545 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P b026a20da9724c08ac646153e4a0385b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.385046 5566 mvcc.cc:205] Tried to move back new op lower bound from 7333856667160018944 to 7333856667112890368. Current Snapshot: MvccSnapshot[applied={T|T < 7333856667160018944}]
I20260927 06:59:10.385121 5543 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.385224 5543 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P eb0f453aa3ea466591ad363641fc79d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 06:59:10.384531 5544 tablet_replica.cc:439] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "af6dacf92b63475fa725d864a7a708d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } } }
I20260927 06:59:10.385578 5544 tablet_replica.cc:442] T f053b6544ee54b7f83b5f3848397b9d7 P af6dacf92b63475fa725d864a7a708d8: This TxnStatusTablet replica's current role is: LEADER
I20260927 06:59:10.385794 5565 mvcc.cc:205] Tried to move back new op lower bound from 7333856667160018944 to 7333856667112890368. Current Snapshot: MvccSnapshot[applied={T|T < 7333856667160018944}]
I20260927 06:59:10.390385 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.387641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.3.57.1:32855)
0927 06:59:10.387685 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.387688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.387697 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:10.387776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.387777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.387792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.387849 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:10.387978 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.388444 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.389571 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.389779 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.389844 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.390057 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.390168 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.390239 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.390312 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.390333 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:10.395668 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.392880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:49837 (local address 127.3.57.2:46569)
0927 06:59:10.392948 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.392950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.393047 (+ 97us) server_negotiation.cc:407] Connection header received
0927 06:59:10.393088 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.393089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.393100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.393164 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:10.393256 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.393984 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.394807 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.395133 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.395203 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.395391 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.395425 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.395476 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.395523 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.395585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 06:59:10.395685 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.392836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.3.57.1:49837)
0927 06:59:10.392976 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.392987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.393044 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.393165 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.393166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.393167 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:10.393218 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.393220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.394065 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.394066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.394724 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.394727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.395207 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.395208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.395280 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.395295 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.395484 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.395486 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.395520 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.395611 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 06:59:10.398095 5468 raft_consensus.cc:1261] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Refusing update from remote peer eb0f453aa3ea466591ad363641fc79d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.398113 5326 raft_consensus.cc:1261] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Refusing update from remote peer eb0f453aa3ea466591ad363641fc79d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.398353 5543 consensus_queue.cc:1035] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:10.398514 5547 consensus_queue.cc:1035] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:10.406885 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.404276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:48621 (local address 127.3.57.1:32855)
0927 06:59:10.404309 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:10.404311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:10.404362 (+ 51us) server_negotiation.cc:407] Connection header received
0927 06:59:10.404425 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:10.404426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:10.404434 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:10.404477 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:10.404586 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.405045 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.406159 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.406379 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.406439 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.406658 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.406696 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.406755 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:10.406813 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 06:59:10.406832 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 06:59:10.406934 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:10.404248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:32855 (local address 127.3.57.2:48621)
0927 06:59:10.404309 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:10.404321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:10.404370 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:10.404482 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:10.404482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:10.404484 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:10.404535 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.404537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.405108 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.405109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.406086 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:10.406089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:10.406458 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:10.406459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:10.406528 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:10.406544 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:10.406757 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:10.406759 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:10.406812 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 06:59:10.406859 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 06:59:10.416839 5030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:10.416951 5030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:10.417783 5030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Sending DeleteTablet for 3 replicas of tablet 06b3a52e178543678918c6cc03c505c6
I20260927 06:59:10.418114 5448 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3a52e178543678918c6cc03c505c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:10 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260927 06:59:10.418453 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3a52e178543678918c6cc03c505c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:10 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:46938
I20260927 06:59:10.418622 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet 06b3a52e178543678918c6cc03c505c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:10 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57014
I20260927 06:59:10.418744 5030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:10.418882 5030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:10.419127 5583 tablet_replica.cc:330] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: stopping tablet replica
I20260927 06:59:10.419276 5583 raft_consensus.cc:2229] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:10.419416 5583 raft_consensus.cc:2258] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:10.419936 5583 ts_tablet_manager.cc:1905] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:10.420251 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 50408d33a6d64d02b27402d58c6eb11c (DEFAULT_TABLE table=test-table [id=9422e2e896604ab0a5737132214640ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.420250 5448 tablet_service.cc:1467] Processing CreateTablet for tablet 50408d33a6d64d02b27402d58c6eb11c (DEFAULT_TABLE table=test-table [id=9422e2e896604ab0a5737132214640ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.420357 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50408d33a6d64d02b27402d58c6eb11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.420369 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50408d33a6d64d02b27402d58c6eb11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.421627 5540 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Bootstrap starting.
I20260927 06:59:10.421728 5539 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Bootstrap starting.
I20260927 06:59:10.422083 5540 tablet_bootstrap.cc:654] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.422127 5539 tablet_bootstrap.cc:654] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.422690 5168 tablet_service.cc:1467] Processing CreateTablet for tablet 50408d33a6d64d02b27402d58c6eb11c (DEFAULT_TABLE table=test-table [id=9422e2e896604ab0a5737132214640ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:10.422813 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50408d33a6d64d02b27402d58c6eb11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:10.423657 5583 ts_tablet_manager.cc:1918] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:10.423771 5583 log.cc:1196] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/06b3a52e178543678918c6cc03c505c6
I20260927 06:59:10.424093 5583 ts_tablet_manager.cc:1939] T 06b3a52e178543678918c6cc03c505c6 P af6dacf92b63475fa725d864a7a708d8: Deleting consensus metadata
I20260927 06:59:10.424163 5537 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Bootstrap starting.
I20260927 06:59:10.424571 5537 tablet_bootstrap.cc:654] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:10.424588 5016 catalog_manager.cc:4841] TS af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569): tablet 06b3a52e178543678918c6cc03c505c6 (table test-table [id=2876503384aa43f29f5de250f58d3d46]) successfully deleted
I20260927 06:59:10.425057 5540 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: No bootstrap required, opened a new log
I20260927 06:59:10.425132 5540 ts_tablet_manager.cc:1397] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 06:59:10.425293 5540 raft_consensus.cc:348] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.425376 5540 raft_consensus.cc:374] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.425405 5540 raft_consensus.cc:729] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.425446 5540 consensus_queue.cc:260] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.425554 5540 ts_tablet_manager.cc:1428] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:10.425670 5537 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: No bootstrap required, opened a new log
I20260927 06:59:10.425707 5537 ts_tablet_manager.cc:1397] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:10.425819 5539 tablet_bootstrap.cc:492] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: No bootstrap required, opened a new log
I20260927 06:59:10.425863 5539 ts_tablet_manager.cc:1397] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 06:59:10.425856 5537 raft_consensus.cc:348] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.425920 5537 raft_consensus.cc:374] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.425940 5537 raft_consensus.cc:729] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.425992 5537 consensus_queue.cc:260] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.425993 5539 raft_consensus.cc:348] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.426076 5539 raft_consensus.cc:374] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:10.426102 5539 raft_consensus.cc:729] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Initialized, Role: FOLLOWER
I20260927 06:59:10.426095 5537 ts_tablet_manager.cc:1428] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:10.426141 5539 consensus_queue.cc:260] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.426245 5539 ts_tablet_manager.cc:1428] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:10.427039 5582 tablet_replica.cc:330] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: stopping tablet replica
I20260927 06:59:10.427160 5582 raft_consensus.cc:2229] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:10.427330 5584 tablet_replica.cc:330] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: stopping tablet replica
I20260927 06:59:10.427419 5584 raft_consensus.cc:2229] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:10.427516 5584 raft_consensus.cc:2258] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:10.427228 5582 raft_consensus.cc:2258] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:10.427958 5584 ts_tablet_manager.cc:1905] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:10.428133 5582 ts_tablet_manager.cc:1905] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:10.429675 5584 ts_tablet_manager.cc:1918] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:10.429797 5584 log.cc:1196] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/06b3a52e178543678918c6cc03c505c6
I20260927 06:59:10.429965 5582 ts_tablet_manager.cc:1918] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:10.430037 5582 log.cc:1196] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/06b3a52e178543678918c6cc03c505c6
I20260927 06:59:10.430214 5584 ts_tablet_manager.cc:1939] T 06b3a52e178543678918c6cc03c505c6 P eb0f453aa3ea466591ad363641fc79d0: Deleting consensus metadata
I20260927 06:59:10.430327 5582 ts_tablet_manager.cc:1939] T 06b3a52e178543678918c6cc03c505c6 P b026a20da9724c08ac646153e4a0385b: Deleting consensus metadata
I20260927 06:59:10.430706 5015 catalog_manager.cc:4841] TS eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855): tablet 06b3a52e178543678918c6cc03c505c6 (table test-table [id=2876503384aa43f29f5de250f58d3d46]) successfully deleted
I20260927 06:59:10.430819 5017 catalog_manager.cc:4841] TS b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321): tablet 06b3a52e178543678918c6cc03c505c6 (table test-table [id=2876503384aa43f29f5de250f58d3d46]) successfully deleted
I20260927 06:59:10.533478 5543 raft_consensus.cc:482] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:10.533561 5543 raft_consensus.cc:504] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.533751 5543 leader_election.cc:288] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569), b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321)
I20260927 06:59:10.533970 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50408d33a6d64d02b27402d58c6eb11c" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8" is_pre_election: true
I20260927 06:59:10.534003 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50408d33a6d64d02b27402d58c6eb11c" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b" is_pre_election: true
I20260927 06:59:10.534083 5468 raft_consensus.cc:2454] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 0.
I20260927 06:59:10.534091 5326 raft_consensus.cc:2454] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 0.
I20260927 06:59:10.534258 5120 leader_election.cc:302] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6dacf92b63475fa725d864a7a708d8, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:10.534363 5543 raft_consensus.cc:2781] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:10.534410 5543 raft_consensus.cc:482] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:10.534444 5543 raft_consensus.cc:3037] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.535094 5543 raft_consensus.cc:504] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.535228 5543 leader_election.cc:288] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 election: Requested vote from peers af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569), b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321)
I20260927 06:59:10.535394 5468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50408d33a6d64d02b27402d58c6eb11c" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b026a20da9724c08ac646153e4a0385b"
I20260927 06:59:10.535406 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50408d33a6d64d02b27402d58c6eb11c" candidate_uuid: "eb0f453aa3ea466591ad363641fc79d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8"
I20260927 06:59:10.535465 5326 raft_consensus.cc:3037] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.535486 5468 raft_consensus.cc:3037] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:10.536084 5468 raft_consensus.cc:2454] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 1.
I20260927 06:59:10.536103 5326 raft_consensus.cc:2454] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0f453aa3ea466591ad363641fc79d0 in term 1.
I20260927 06:59:10.536269 5121 leader_election.cc:302] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b026a20da9724c08ac646153e4a0385b, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:10.536409 5543 raft_consensus.cc:2781] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:10.536494 5543 raft_consensus.cc:686] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 1 LEADER]: Becoming Leader. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Running, Role: LEADER
I20260927 06:59:10.536551 5543 consensus_queue.cc:237] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:10.537141 5031 catalog_manager.cc:5495] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "eb0f453aa3ea466591ad363641fc79d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } health_report { overall_health: UNKNOWN } } }
I20260927 06:59:10.545437 5326 raft_consensus.cc:1261] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Refusing update from remote peer eb0f453aa3ea466591ad363641fc79d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.545450 5468 raft_consensus.cc:1261] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Refusing update from remote peer eb0f453aa3ea466591ad363641fc79d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:10.545671 5547 consensus_queue.cc:1035] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:10.545800 5543 consensus_queue.cc:1035] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 06:59:10.550277 5565 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 06:59:12.067246 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:12.067343 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:12.068256 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Sending DeleteTablet for 3 replicas of tablet 50408d33a6d64d02b27402d58c6eb11c
I20260927 06:59:12.068522 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 50408d33a6d64d02b27402d58c6eb11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:46938
I20260927 06:59:12.068535 5448 tablet_service.cc:1514] Processing DeleteTablet for tablet 50408d33a6d64d02b27402d58c6eb11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260927 06:59:12.068691 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet 50408d33a6d64d02b27402d58c6eb11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57014
I20260927 06:59:12.068729 5614 tablet_replica.cc:330] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: stopping tablet replica
I20260927 06:59:12.068805 5614 raft_consensus.cc:2229] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:12.068836 5614 raft_consensus.cc:2258] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.068883 5616 tablet_replica.cc:330] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: stopping tablet replica
I20260927 06:59:12.068925 5615 tablet_replica.cc:330] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: stopping tablet replica
I20260927 06:59:12.068933 5616 raft_consensus.cc:2229] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:12.068969 5615 raft_consensus.cc:2229] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:12.068981 5616 raft_consensus.cc:2258] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.069018 5615 raft_consensus.cc:2258] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.069267 5616 ts_tablet_manager.cc:1905] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.070711 5614 ts_tablet_manager.cc:1905] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.071063 5616 ts_tablet_manager.cc:1918] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:59:12.071143 5616 log.cc:1196] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/50408d33a6d64d02b27402d58c6eb11c
I20260927 06:59:12.071376 5616 ts_tablet_manager.cc:1939] T 50408d33a6d64d02b27402d58c6eb11c P eb0f453aa3ea466591ad363641fc79d0: Deleting consensus metadata
I20260927 06:59:12.071727 5015 catalog_manager.cc:4841] TS eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855): tablet 50408d33a6d64d02b27402d58c6eb11c (table test-table [id=9422e2e896604ab0a5737132214640ed]) successfully deleted
I20260927 06:59:12.072412 5614 ts_tablet_manager.cc:1918] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:59:12.072472 5614 log.cc:1196] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/50408d33a6d64d02b27402d58c6eb11c
I20260927 06:59:12.072633 5614 ts_tablet_manager.cc:1939] T 50408d33a6d64d02b27402d58c6eb11c P af6dacf92b63475fa725d864a7a708d8: Deleting consensus metadata
I20260927 06:59:12.072821 5615 ts_tablet_manager.cc:1905] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.073381 5016 catalog_manager.cc:4841] TS af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569): tablet 50408d33a6d64d02b27402d58c6eb11c (table test-table [id=9422e2e896604ab0a5737132214640ed]) successfully deleted
I20260927 06:59:12.074498 5615 ts_tablet_manager.cc:1918] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:59:12.074555 5615 log.cc:1196] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/50408d33a6d64d02b27402d58c6eb11c
I20260927 06:59:12.074707 5615 ts_tablet_manager.cc:1939] T 50408d33a6d64d02b27402d58c6eb11c P b026a20da9724c08ac646153e4a0385b: Deleting consensus metadata
I20260927 06:59:12.075302 5017 catalog_manager.cc:4841] TS b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321): tablet 50408d33a6d64d02b27402d58c6eb11c (table test-table [id=9422e2e896604ab0a5737132214640ed]) successfully deleted
Sep 27 06:59:12 dist-test-slave-mk3h krb5kdc[4986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492352, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 06:59:12 dist-test-slave-mk3h krb5kdc[4986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492352, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:12.095698 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.088601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.3.57.62:34411)
0927 06:59:12.088773 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.088775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.088784 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:12.088853 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.088854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.088864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.088935 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:12.089482 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.089990 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.090726 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.090914 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.090965 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.093225 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:12.093226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:12.093227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:12.093233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:12.094517 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:12.094830 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:12.094831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:12.094831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:12.094890 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:12.095120 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:12.095121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:12.095121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:12.095245 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:12.095328 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.095442 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.095508 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260927 06:59:12.097211 5029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:12.097433 5029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:12.098888 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 (DEFAULT_TABLE table=test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.098897 5448 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 (DEFAULT_TABLE table=test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.098899 5168 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 (DEFAULT_TABLE table=test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.099023 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed2025cb28f4add9f5d0cf210565f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.099023 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed2025cb28f4add9f5d0cf210565f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.099046 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed2025cb28f4add9f5d0cf210565f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.100184 5625 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Bootstrap starting.
I20260927 06:59:12.100392 5626 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Bootstrap starting.
I20260927 06:59:12.100606 5625 tablet_bootstrap.cc:654] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.100782 5626 tablet_bootstrap.cc:654] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.101130 5627 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Bootstrap starting.
I20260927 06:59:12.101276 5625 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: No bootstrap required, opened a new log
I20260927 06:59:12.101336 5625 ts_tablet_manager.cc:1397] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:12.101450 5625 raft_consensus.cc:348] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.101528 5627 tablet_bootstrap.cc:654] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.101536 5625 raft_consensus.cc:374] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.101559 5625 raft_consensus.cc:729] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.101619 5625 consensus_queue.cc:260] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.101756 5625 ts_tablet_manager.cc:1428] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:12.101923 5626 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: No bootstrap required, opened a new log
I20260927 06:59:12.102003 5626 ts_tablet_manager.cc:1397] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:12.102137 5626 raft_consensus.cc:348] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.102404 5627 tablet_bootstrap.cc:492] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: No bootstrap required, opened a new log
I20260927 06:59:12.102460 5627 ts_tablet_manager.cc:1397] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:12.102595 5627 raft_consensus.cc:348] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.102669 5627 raft_consensus.cc:374] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.102691 5627 raft_consensus.cc:729] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.102744 5627 consensus_queue.cc:260] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.102841 5627 ts_tablet_manager.cc:1428] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:12.102897 5626 raft_consensus.cc:374] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.102937 5626 raft_consensus.cc:729] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.102988 5626 consensus_queue.cc:260] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.103156 5626 ts_tablet_manager.cc:1428] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 06:59:12.197449 5631 raft_consensus.cc:482] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:12.197556 5631 raft_consensus.cc:504] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.197907 5631 leader_election.cc:288] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:12.200925 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.198207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:44959 (local address 127.3.57.2:46569)
0927 06:59:12.198400 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.198402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.198413 (+ 11us) server_negotiation.cc:407] Connection header received
0927 06:59:12.198441 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.198442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.198458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.198537 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:12.198676 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.199190 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200203 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.200575 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.200678 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.200806 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.200849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
I20260927 06:59:12.200927 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.198116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.3.57.3:44959)
0927 06:59:12.198296 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.198308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:12.198369 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:12.198541 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:12.198541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:12.198544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:12.198600 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.198602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.199276 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.199276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.200110 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.200112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.200668 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.200734 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.200738 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:12.200808 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 06:59:12.200849 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260927 06:59:12.201035 5635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.198273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:41281 (local address 127.3.57.1:32855)
0927 06:59:12.198412 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.198414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.198447 (+ 33us) server_negotiation.cc:407] Connection header received
0927 06:59:12.198535 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.198536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.198552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.198633 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:12.198808 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.199300 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200298 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.200662 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.200755 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.200912 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.200955 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260927 06:59:12.201061 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.198184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:32855 (local address 127.3.57.3:41281)
0927 06:59:12.198382 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.198393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:12.198457 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:12.198635 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:12.198636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:12.198637 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:12.198707 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.198709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.199385 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.199386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.200195 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.200197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.200744 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.200751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.200821 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.200831 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 06:59:12.200894 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 06:59:12.200936 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":13,"thread_start_us":114,"threads_started":1}
I20260927 06:59:12.201257 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed2025cb28f4add9f5d0cf210565f13" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0" is_pre_election: true
I20260927 06:59:12.201355 5188 raft_consensus.cc:2454] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 0.
I20260927 06:59:12.201347 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed2025cb28f4add9f5d0cf210565f13" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8" is_pre_election: true
I20260927 06:59:12.201432 5326 raft_consensus.cc:2454] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 0.
I20260927 06:59:12.201552 5399 leader_election.cc:302] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b026a20da9724c08ac646153e4a0385b, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:12.201699 5631 raft_consensus.cc:2781] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:12.201749 5631 raft_consensus.cc:482] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:12.201781 5631 raft_consensus.cc:3037] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.202467 5631 raft_consensus.cc:504] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.202616 5631 leader_election.cc:288] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 election: Requested vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:12.202768 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed2025cb28f4add9f5d0cf210565f13" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8"
I20260927 06:59:12.202850 5326 raft_consensus.cc:3037] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.202811 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed2025cb28f4add9f5d0cf210565f13" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0"
I20260927 06:59:12.202903 5188 raft_consensus.cc:3037] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.203447 5326 raft_consensus.cc:2454] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 1.
I20260927 06:59:12.203585 5188 raft_consensus.cc:2454] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 1.
I20260927 06:59:12.203647 5400 leader_election.cc:302] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6dacf92b63475fa725d864a7a708d8, b026a20da9724c08ac646153e4a0385b; no voters:
I20260927 06:59:12.203722 5631 raft_consensus.cc:2781] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:12.203860 5631 raft_consensus.cc:686] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 1 LEADER]: Becoming Leader. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Running, Role: LEADER
I20260927 06:59:12.203955 5631 consensus_queue.cc:237] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.204728 5031 catalog_manager.cc:5495] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b reported cstate change: term changed from 0 to 1, leader changed from <none> to b026a20da9724c08ac646153e4a0385b (127.3.57.3). New cstate: current_term: 1 leader_uuid: "b026a20da9724c08ac646153e4a0385b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:12.224963 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.221974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.3.57.3:33321)
0927 06:59:12.222148 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.222149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.222155 (+ 6us) server_negotiation.cc:407] Connection header received
0927 06:59:12.222189 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.222190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.222208 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.222323 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:12.222439 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.222959 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.223850 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.224242 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.224320 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.224560 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:12.224707 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:12.224768 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.224830 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.224848 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260927 06:59:12.229636 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.227252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:57199 (local address 127.3.57.2:46569)
0927 06:59:12.227331 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.227333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.227436 (+ 103us) server_negotiation.cc:407] Connection header received
0927 06:59:12.227500 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.227501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.227514 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.227595 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:12.227729 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.228179 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.228938 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.229148 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.229223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.229383 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:12.229423 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:12.229482 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.229543 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.229563 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 06:59:12.229666 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.227204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46569 (local address 127.3.57.3:57199)
0927 06:59:12.227368 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.227383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:12.227450 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:12.227602 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:12.227603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:12.227604 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:12.227660 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.227661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.228260 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.228261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.228852 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.228854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.229222 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.229223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.229288 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.229306 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:12.229484 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:12.229487 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:12.229538 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 06:59:12.229582 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260927 06:59:12.231796 5326 raft_consensus.cc:1261] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Refusing update from remote peer b026a20da9724c08ac646153e4a0385b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:12.231806 5188 raft_consensus.cc:1261] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Refusing update from remote peer b026a20da9724c08ac646153e4a0385b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:12.232106 5636 consensus_queue.cc:1035] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:12.232239 5631 consensus_queue.cc:1035] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:12.238821 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.236559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:43007 (local address 127.3.57.3:33321)
0927 06:59:12.236595 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:12.236597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:12.236732 (+ 135us) server_negotiation.cc:407] Connection header received
0927 06:59:12.236792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:12.236795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:12.236812 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:12.236851 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:12.236926 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.237363 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.238059 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.238366 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.238422 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.238574 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:12.238607 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:12.238659 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:12.238724 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 06:59:12.238746 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 06:59:12.238871 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:12.236532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33321 (local address 127.3.57.2:43007)
0927 06:59:12.236659 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:12.236671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:12.236727 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:12.236852 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:12.236853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:12.236854 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 06:59:12.236893 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.236894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.237420 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.237420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.237973 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:12.237975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:12.238438 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:12.238439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:12.238502 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:12.238507 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 06:59:12.238661 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 06:59:12.238663 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:12.238721 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 06:59:12.238778 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":88,"threads_started":1}
I20260927 06:59:12.245249 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:12.245378 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:12.246040 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Sending DeleteTablet for 3 replicas of tablet 9ed2025cb28f4add9f5d0cf210565f13
I20260927 06:59:12.246295 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57014
I20260927 06:59:12.246356 5448 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260927 06:59:12.246416 5616 tablet_replica.cc:330] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: stopping tablet replica
I20260927 06:59:12.246459 5615 tablet_replica.cc:330] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: stopping tablet replica
I20260927 06:59:12.246469 5616 raft_consensus.cc:2229] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:12.246505 5616 raft_consensus.cc:2258] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.246506 5615 raft_consensus.cc:2229] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:12.246582 5615 raft_consensus.cc:2258] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.246644 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed2025cb28f4add9f5d0cf210565f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:46938
I20260927 06:59:12.246604 5029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 06:59:12.246739 5614 tablet_replica.cc:330] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: stopping tablet replica
W20260927 06:59:12.246743 5029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:12.246783 5614 raft_consensus.cc:2229] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:12.246829 5614 raft_consensus.cc:2258] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:12.247174 5614 ts_tablet_manager.cc:1905] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.247180 5616 ts_tablet_manager.cc:1905] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.247444 5615 ts_tablet_manager.cc:1905] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:12.248821 5616 ts_tablet_manager.cc:1918] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:12.248824 5614 ts_tablet_manager.cc:1918] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:12.248875 5616 log.cc:1196] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/9ed2025cb28f4add9f5d0cf210565f13
I20260927 06:59:12.248879 5614 log.cc:1196] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/9ed2025cb28f4add9f5d0cf210565f13
I20260927 06:59:12.248940 5615 ts_tablet_manager.cc:1918] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 06:59:12.248989 5615 log.cc:1196] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/9ed2025cb28f4add9f5d0cf210565f13
I20260927 06:59:12.249104 5616 ts_tablet_manager.cc:1939] T 9ed2025cb28f4add9f5d0cf210565f13 P eb0f453aa3ea466591ad363641fc79d0: Deleting consensus metadata
I20260927 06:59:12.249109 5614 ts_tablet_manager.cc:1939] T 9ed2025cb28f4add9f5d0cf210565f13 P af6dacf92b63475fa725d864a7a708d8: Deleting consensus metadata
I20260927 06:59:12.249186 5615 ts_tablet_manager.cc:1939] T 9ed2025cb28f4add9f5d0cf210565f13 P b026a20da9724c08ac646153e4a0385b: Deleting consensus metadata
I20260927 06:59:12.249576 5015 catalog_manager.cc:4841] TS eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855): tablet 9ed2025cb28f4add9f5d0cf210565f13 (table test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]) successfully deleted
I20260927 06:59:12.249728 5016 catalog_manager.cc:4841] TS af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569): tablet 9ed2025cb28f4add9f5d0cf210565f13 (table test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]) successfully deleted
I20260927 06:59:12.249996 5168 tablet_service.cc:1467] Processing CreateTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b (DEFAULT_TABLE table=test-table [id=97fafabcba3d4d0986cc2e62b2becb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.250010 5308 tablet_service.cc:1467] Processing CreateTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b (DEFAULT_TABLE table=test-table [id=97fafabcba3d4d0986cc2e62b2becb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.250100 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13f20f7caf2446992bb3ce9ef87f68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.250115 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13f20f7caf2446992bb3ce9ef87f68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.250227 5448 tablet_service.cc:1467] Processing CreateTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b (DEFAULT_TABLE table=test-table [id=97fafabcba3d4d0986cc2e62b2becb0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:12.250309 5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13f20f7caf2446992bb3ce9ef87f68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:12.251370 5627 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Bootstrap starting.
I20260927 06:59:12.251423 5626 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Bootstrap starting.
I20260927 06:59:12.251447 5625 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0: Bootstrap starting.
I20260927 06:59:12.251731 5627 tablet_bootstrap.cc:654] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.251740 5625 tablet_bootstrap.cc:654] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.251892 5626 tablet_bootstrap.cc:654] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:12.252592 5627 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: No bootstrap required, opened a new log
I20260927 06:59:12.252648 5627 ts_tablet_manager.cc:1397] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:12.252674 5625 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0: No bootstrap required, opened a new log
I20260927 06:59:12.252723 5625 ts_tablet_manager.cc:1397] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:12.252753 5627 raft_consensus.cc:348] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.252780 5626 tablet_bootstrap.cc:492] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: No bootstrap required, opened a new log
I20260927 06:59:12.252801 5627 raft_consensus.cc:374] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.252815 5627 raft_consensus.cc:729] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6dacf92b63475fa725d864a7a708d8, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.252818 5626 ts_tablet_manager.cc:1397] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:12.252832 5625 raft_consensus.cc:348] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.252876 5627 consensus_queue.cc:260] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.252903 5625 raft_consensus.cc:374] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.252929 5625 raft_consensus.cc:729] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f453aa3ea466591ad363641fc79d0, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.252969 5627 ts_tablet_manager.cc:1428] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:12.252974 5625 consensus_queue.cc:260] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.252983 5626 raft_consensus.cc:348] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.253043 5626 raft_consensus.cc:374] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:12.253063 5626 raft_consensus.cc:729] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Initialized, Role: FOLLOWER
I20260927 06:59:12.253063 5625 ts_tablet_manager.cc:1428] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:12.253106 5626 consensus_queue.cc:260] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.253226 5626 ts_tablet_manager.cc:1428] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:12.253605 5017 catalog_manager.cc:4841] TS b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321): tablet 9ed2025cb28f4add9f5d0cf210565f13 (table test-table [id=e03694b8e25549cd8f71a93e6f2ee89a]) successfully deleted
I20260927 06:59:12.352963 5648 raft_consensus.cc:482] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:12.353084 5648 raft_consensus.cc:504] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.353348 5648 leader_election.cc:288] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:12.353497 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13f20f7caf2446992bb3ce9ef87f68b" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0" is_pre_election: true
I20260927 06:59:12.353492 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13f20f7caf2446992bb3ce9ef87f68b" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8" is_pre_election: true
I20260927 06:59:12.353590 5188 raft_consensus.cc:2454] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 0.
I20260927 06:59:12.353628 5326 raft_consensus.cc:2454] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 0.
I20260927 06:59:12.353775 5399 leader_election.cc:302] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b026a20da9724c08ac646153e4a0385b, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:12.353875 5648 raft_consensus.cc:2781] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:12.353909 5648 raft_consensus.cc:482] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:12.353929 5648 raft_consensus.cc:3037] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.355207 5648 raft_consensus.cc:504] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.355458 5648 leader_election.cc:288] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 election: Requested vote from peers eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855), af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569)
I20260927 06:59:12.355569 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13f20f7caf2446992bb3ce9ef87f68b" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f453aa3ea466591ad363641fc79d0"
I20260927 06:59:12.355697 5188 raft_consensus.cc:3037] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.355688 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13f20f7caf2446992bb3ce9ef87f68b" candidate_uuid: "b026a20da9724c08ac646153e4a0385b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6dacf92b63475fa725d864a7a708d8"
I20260927 06:59:12.355767 5326 raft_consensus.cc:3037] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:12.356518 5188 raft_consensus.cc:2454] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 1.
I20260927 06:59:12.356631 5326 raft_consensus.cc:2454] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b026a20da9724c08ac646153e4a0385b in term 1.
I20260927 06:59:12.356742 5399 leader_election.cc:302] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b026a20da9724c08ac646153e4a0385b, eb0f453aa3ea466591ad363641fc79d0; no voters:
I20260927 06:59:12.356866 5648 raft_consensus.cc:2781] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:12.356953 5648 raft_consensus.cc:686] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 1 LEADER]: Becoming Leader. State: Replica: b026a20da9724c08ac646153e4a0385b, State: Running, Role: LEADER
I20260927 06:59:12.357012 5648 consensus_queue.cc:237] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } }
I20260927 06:59:12.357635 5029 catalog_manager.cc:5495] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b reported cstate change: term changed from 0 to 1, leader changed from <none> to b026a20da9724c08ac646153e4a0385b (127.3.57.3). New cstate: current_term: 1 leader_uuid: "b026a20da9724c08ac646153e4a0385b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b026a20da9724c08ac646153e4a0385b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33321 } health_report { overall_health: HEALTHY } } }
I20260927 06:59:12.375659 5326 raft_consensus.cc:1261] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Refusing update from remote peer b026a20da9724c08ac646153e4a0385b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:12.375699 5188 raft_consensus.cc:1261] T f13f20f7caf2446992bb3ce9ef87f68b P eb0f453aa3ea466591ad363641fc79d0 [term 1 FOLLOWER]: Refusing update from remote peer b026a20da9724c08ac646153e4a0385b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:12.375944 5648 consensus_queue.cc:1035] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0f453aa3ea466591ad363641fc79d0" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:12.376075 5636 consensus_queue.cc:1035] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6dacf92b63475fa725d864a7a708d8" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 06:59:12.380666 5598 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 06:59:13.907269 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:13.907393 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:13.908399 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c5b9837a3bc4267846846987e90dfeb: Sending DeleteTablet for 3 replicas of tablet f13f20f7caf2446992bb3ce9ef87f68b
I20260927 06:59:13.908624 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:13 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:46938
I20260927 06:59:13.908658 5448 tablet_service.cc:1514] Processing DeleteTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:13 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260927 06:59:13.908819 5168 tablet_service.cc:1514] Processing DeleteTablet for tablet f13f20f7caf2446992bb3ce9ef87f68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:13 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57014
I20260927 06:59:13.908833 5677 tablet_replica.cc:330] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: stopping tablet replica
I20260927 06:59:13.908902 5677 raft_consensus.cc:2229] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:13.908944 5677 raft_consensus.cc:2258] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:13.908928 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5103
I20260927 06:59:13.909708 5677 ts_tablet_manager.cc:1905] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:13.910368 5678 tablet_replica.cc:330] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: stopping tablet replica
I20260927 06:59:13.910454 5678 raft_consensus.cc:2229] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:13.910530 5678 raft_consensus.cc:2258] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:13.910758 5678 ts_tablet_manager.cc:1905] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:13.911763 5677 ts_tablet_manager.cc:1918] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:59:13.911826 5677 log.cc:1196] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/f13f20f7caf2446992bb3ce9ef87f68b
I20260927 06:59:13.912063 5677 ts_tablet_manager.cc:1939] T f13f20f7caf2446992bb3ce9ef87f68b P af6dacf92b63475fa725d864a7a708d8: Deleting consensus metadata
I20260927 06:59:13.912078 5678 ts_tablet_manager.cc:1918] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 06:59:13.912144 5678 log.cc:1196] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/f13f20f7caf2446992bb3ce9ef87f68b
I20260927 06:59:13.912385 5016 catalog_manager.cc:4841] TS af6dacf92b63475fa725d864a7a708d8 (127.3.57.2:46569): tablet f13f20f7caf2446992bb3ce9ef87f68b (table test-table [id=97fafabcba3d4d0986cc2e62b2becb0f]) successfully deleted
I20260927 06:59:13.912470 5678 ts_tablet_manager.cc:1939] T f13f20f7caf2446992bb3ce9ef87f68b P b026a20da9724c08ac646153e4a0385b: Deleting consensus metadata
I20260927 06:59:13.912856 5017 catalog_manager.cc:4841] TS b026a20da9724c08ac646153e4a0385b (127.3.57.3:33321): tablet f13f20f7caf2446992bb3ce9ef87f68b (table test-table [id=97fafabcba3d4d0986cc2e62b2becb0f]) successfully deleted
W20260927 06:59:13.916316 5015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:32855 (error 108)
I20260927 06:59:13.916625 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5243
I20260927 06:59:13.917382 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:13.916700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:32855 (local address 127.0.0.1:57032)
0927 06:59:13.917305 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:13.917344 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":58,"threads_started":1}
W20260927 06:59:13.917522 5015 catalog_manager.cc:4584] TS eb0f453aa3ea466591ad363641fc79d0 (127.3.57.1:32855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f13f20f7caf2446992bb3ce9ef87f68b: Network error: Client connection negotiation failed: client connection to 127.3.57.1:32855: connect: Connection refused (error 111)
I20260927 06:59:13.923306 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5383
I20260927 06:59:13.928916 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 4999
2026-09-27T06:59:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5621 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:13 dist-test-slave-mk3h krb5kdc[5684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:13 dist-test-slave-mk3h krb5kdc[5684](info): set up 2 sockets
Sep 27 06:59:13 dist-test-slave-mk3h krb5kdc[5684](info): commencing operation
krb5kdc: starting...
W20260927 06:59:14.963644 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:15.098330 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41929
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:15.177107 5697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:15.177321 5697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:15.177357 5697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:15.178447 5697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:15.178509 5697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:15.178539 5697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:15.178565 5697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:15.178614 5697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:15.180058 5697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:15.180315 5697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:15.180544 5697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.185139 5703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.185230 5697 server_base.cc:1034] running on GCE node
W20260927 06:59:15.185302 5702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.185146 5705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.185590 5697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.185861 5697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.187018 5697 hybrid_clock.cc:648] HybridClock initialized: now 1790492355186994 us; error 37 us; skew 500 ppm
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.189335 5697 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:15.189718 5697 webserver.cc:460] Webserver started at http://127.3.57.62:39797/ using document root <none> and password file <none>
I20260927 06:59:15.189925 5697 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.190006 5697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.190156 5697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.191133 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "fb0b507dcfca4aa18b29b42b5428aea4"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "588e5a380c6fd53eaeb45a4e88da1100"
server_key_iv: "8904a84dbbe8315c96978ef11bb5a0d2"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.191485 5697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "fb0b507dcfca4aa18b29b42b5428aea4"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "588e5a380c6fd53eaeb45a4e88da1100"
server_key_iv: "8904a84dbbe8315c96978ef11bb5a0d2"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.192765 5697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:15.193488 5712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.193660 5697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:15.193758 5697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "fb0b507dcfca4aa18b29b42b5428aea4"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "588e5a380c6fd53eaeb45a4e88da1100"
server_key_iv: "8904a84dbbe8315c96978ef11bb5a0d2"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.193838 5697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.211818 5697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.213519 5697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.213672 5697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.217548 5697 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41929
I20260927 06:59:15.217584 5764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41929 every 8 connection(s)
I20260927 06:59:15.217941 5697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:15.218801 5765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:15.221102 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: Bootstrap starting.
I20260927 06:59:15.221673 5765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:15.221976 5765 log.cc:824] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:15.222726 5765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: No bootstrap required, opened a new log
I20260927 06:59:15.222759 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5697
I20260927 06:59:15.222833 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:15.223002 3300 external_mini_cluster.cc:1428] Setting key 72a470122645ff14849e7064a2f03b2a
I20260927 06:59:15.225272 5765 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } }
I20260927 06:59:15.225404 5765 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:15.225425 5765 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0b507dcfca4aa18b29b42b5428aea4, State: Initialized, Role: FOLLOWER
I20260927 06:59:15.225549 5765 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } }
I20260927 06:59:15.225625 5765 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:15.225648 5765 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:15.225720 5765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:15.226280 5765 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } }
I20260927 06:59:15.226364 5765 leader_election.cc:302] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0b507dcfca4aa18b29b42b5428aea4; no voters:
I20260927 06:59:15.226572 5765 leader_election.cc:288] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:15.226627 5770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:15.226790 5770 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [term 1 LEADER]: Becoming Leader. State: Replica: fb0b507dcfca4aa18b29b42b5428aea4, State: Running, Role: LEADER
I20260927 06:59:15.226852 5765 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:15.226962 5770 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } }
I20260927 06:59:15.227283 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0b507dcfca4aa18b29b42b5428aea4. Latest consensus state: current_term: 1 leader_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } } }
I20260927 06:59:15.227391 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:15.227291 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0b507dcfca4aa18b29b42b5428aea4" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41929 } } }
I20260927 06:59:15.227499 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4 [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:15.228806 5786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:15.228888 5786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:15.228983 5776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:15.229162 5776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:15.231482 5776 catalog_manager.cc:1347] Generated new cluster ID: f7cf5226b4ea4aad991c6f0197742663
I20260927 06:59:15.231572 5776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:15.237022 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.223521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.3.57.62:41929)
0927 06:59:15.223690 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.223697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.223721 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:15.224202 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.224209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.224266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.224373 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.225156 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.226601 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.227716 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.228307 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.228435 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.232285 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.232291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.232291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.232303 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.234747 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.235182 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.235185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.235186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.235294 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.235505 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.235505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.235506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.235920 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.236528 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.236616 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.236777 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260927 06:59:15.249536 5776 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:15.250353 5776 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:15.254050 5776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0b507dcfca4aa18b29b42b5428aea4: Generated new TSK 0
I20260927 06:59:15.254237 5776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:15.298012 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:15.399957 5791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:15.400141 5791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:15.400164 5791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:15.401299 5791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:15.401338 5791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:15.401364 5791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:15.402758 5791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:15.403007 5791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:15.403236 5791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.408784 5797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.409154 5799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.409192 5791 server_base.cc:1034] running on GCE node
W20260927 06:59:15.409211 5796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.409528 5791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.409795 5791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.411034 5791 hybrid_clock.cc:648] HybridClock initialized: now 1790492355410987 us; error 72 us; skew 500 ppm
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.414180 5791 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:15.414645 5791 webserver.cc:460] Webserver started at http://127.3.57.1:46207/ using document root <none> and password file <none>
I20260927 06:59:15.414886 5791 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.414955 5791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.415102 5791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.416453 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "81052f6523c5425fb6c41705f7ad1609"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "1b05b6c5c5e5c3ce55a1345c4320f24a"
server_key_iv: "ab92abab78f1437e68c1e47d6d8e8c3d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.416827 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "81052f6523c5425fb6c41705f7ad1609"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "1b05b6c5c5e5c3ce55a1345c4320f24a"
server_key_iv: "ab92abab78f1437e68c1e47d6d8e8c3d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.418157 5791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 06:59:15.418881 5806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.419073 5791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:15.419175 5791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "81052f6523c5425fb6c41705f7ad1609"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "1b05b6c5c5e5c3ce55a1345c4320f24a"
server_key_iv: "ab92abab78f1437e68c1e47d6d8e8c3d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.419248 5791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.455017 5791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.456962 5791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.457129 5791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.457372 5791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:15.457671 5791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:15.457722 5791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.457762 5791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:15.457800 5791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.463361 5791 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:36533
I20260927 06:59:15.463379 5919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:36533 every 8 connection(s)
I20260927 06:59:15.463832 5791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:15.472627 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.464548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:37861 (local address 127.3.57.62:41929)
0927 06:59:15.464615 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.464617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.464827 (+ 210us) server_negotiation.cc:407] Connection header received
0927 06:59:15.465274 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.465275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.465285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.465337 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.466264 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.466754 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.467849 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.468035 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.468089 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.469960 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.469961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.469962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.469967 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.471353 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.471690 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.471690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.471691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.471741 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.471941 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.471942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.471942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.472066 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.472116 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.472477 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.472563 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:59:15.472777 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.464533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41929 (local address 127.3.57.1:37861)
0927 06:59:15.464703 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.464738 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.465196 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.465404 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.465407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.465968 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:15.466185 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.466190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.466904 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.466905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.467774 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.467776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.468108 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.468109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.468204 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.468614 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:15.468622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:15.469881 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:15.471423 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.471428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.471430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.471642 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.471799 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.471800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.471800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.471899 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.472187 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:15.472193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:15.472327 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.472441 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.472568 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260927 06:59:15.473235 5920 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41929
I20260927 06:59:15.473311 5920 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:15.473476 5920 heartbeater.cc:511] Master 127.3.57.62:41929 requested a full tablet report, sending...
I20260927 06:59:15.473642 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5791
I20260927 06:59:15.473738 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:15.473942 3300 external_mini_cluster.cc:1428] Setting key 312f9cefefcfe9e47f8b1e76690ad860
I20260927 06:59:15.474076 5729 ts_manager.cc:194] Registered new tserver with Master: 81052f6523c5425fb6c41705f7ad1609 (127.3.57.1:36533)
I20260927 06:59:15.475092 5729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:37861
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:15.502910 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:15.583588 5925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:15.583783 5925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:15.583806 5925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:15.584887 5925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:15.584919 5925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:15.584944 5925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:15.586256 5925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:15.586473 5925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:15.586665 5925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.591367 5931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.591490 5925 server_base.cc:1034] running on GCE node
W20260927 06:59:15.591377 5930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.591661 5933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.591843 5925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.592073 5925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.593214 5925 hybrid_clock.cc:648] HybridClock initialized: now 1790492355593200 us; error 27 us; skew 500 ppm
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.595667 5925 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:15.596114 5925 webserver.cc:460] Webserver started at http://127.3.57.2:34311/ using document root <none> and password file <none>
I20260927 06:59:15.596349 5925 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.596416 5925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.596561 5925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.597560 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "9fe231a620f94d4f82b873b0c6c3e99c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "32b1b7606c0ba276aa8bc23e7726d140"
server_key_iv: "77d220aa80ec8cb3debbdb08c330a53d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.597924 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "9fe231a620f94d4f82b873b0c6c3e99c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "32b1b7606c0ba276aa8bc23e7726d140"
server_key_iv: "77d220aa80ec8cb3debbdb08c330a53d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.599166 5925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:15.599896 5940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.600059 5925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.600152 5925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "9fe231a620f94d4f82b873b0c6c3e99c"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "32b1b7606c0ba276aa8bc23e7726d140"
server_key_iv: "77d220aa80ec8cb3debbdb08c330a53d"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.600230 5925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.613919 5925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.616073 5925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.616226 5925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.616434 5925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:15.616726 5925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:15.616779 5925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.616819 5925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:15.616847 5925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.622566 5925 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:41825
I20260927 06:59:15.622606 6053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:41825 every 8 connection(s)
I20260927 06:59:15.622915 5925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:15.627117 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 5925
I20260927 06:59:15.627216 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:15.627409 3300 external_mini_cluster.cc:1428] Setting key 189b9d4a4621885c80a1e8145d0cfb6a
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:15.632611 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.623622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:57941 (local address 127.3.57.62:41929)
0927 06:59:15.623665 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.623667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.623872 (+ 205us) server_negotiation.cc:407] Connection header received
0927 06:59:15.624334 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.624335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.624345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.624394 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.625274 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.625769 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.626809 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.627007 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.627061 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.629717 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.629718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.629719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.629726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.631246 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.631687 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.631688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.631689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.631746 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.631996 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.631997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.631997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.632110 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.632183 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.632448 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.632523 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 06:59:15.632802 6056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.623622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41929 (local address 127.3.57.2:57941)
0927 06:59:15.623757 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.623789 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.624250 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.624441 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.624444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.624962 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:15.625134 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.625138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.625850 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.625850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.626742 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.626745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.627087 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.627088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.627248 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.628084 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:15.628093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:15.629622 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:15.631341 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.631346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.631349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.631608 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.631813 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.631814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.631815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.631924 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.632175 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:15.632181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:15.632316 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.632413 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.632564 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260927 06:59:15.633169 6054 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41929
I20260927 06:59:15.633260 6054 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:15.633446 6054 heartbeater.cc:511] Master 127.3.57.62:41929 requested a full tablet report, sending...
I20260927 06:59:15.633782 5729 ts_manager.cc:194] Registered new tserver with Master: 9fe231a620f94d4f82b873b0c6c3e99c (127.3.57.2:41825)
I20260927 06:59:15.634157 5729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:57941
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:15.654098 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:15.732376 6059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:15.732556 6059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:15.732579 6059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:15.733734 6059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:15.733772 6059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:15.733803 6059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:15.736027 6059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:15.736277 6059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:15.736569 6059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:15.742106 6064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.744596 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:15.744802 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:15.745322 6059 server_base.cc:1034] running on GCE node
I20260927 06:59:15.745536 6059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:15.745792 6059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:15.746956 6059 hybrid_clock.cc:648] HybridClock initialized: now 1790492355746939 us; error 32 us; skew 500 ppm
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:15.749807 6059 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:15.750170 6059 webserver.cc:460] Webserver started at http://127.3.57.3:37667/ using document root <none> and password file <none>
I20260927 06:59:15.750375 6059 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:15.750413 6059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:15.750506 6059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:15.751478 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "6dcfbb40706e4928922a01d68c26770b"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "c5ac0db29bd2509b49442cb4dd1a7110"
server_key_iv: "ed6177d5a3676ff5b45d2d1bdf47c002"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.751883 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "6dcfbb40706e4928922a01d68c26770b"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "c5ac0db29bd2509b49442cb4dd1a7110"
server_key_iv: "ed6177d5a3676ff5b45d2d1bdf47c002"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.753110 6059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:15.753852 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.754030 6059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:15.754120 6059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "6dcfbb40706e4928922a01d68c26770b"
format_stamp: "Formatted at 2026-09-27 06:59:15 on dist-test-slave-mk3h"
server_key: "c5ac0db29bd2509b49442cb4dd1a7110"
server_key_iv: "ed6177d5a3676ff5b45d2d1bdf47c002"
server_key_version: "encryptionkey@0"
I20260927 06:59:15.754197 6059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:15.781481 6059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:15.783272 6059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:15.783375 6059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:15.783591 6059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:15.783910 6059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:15.783942 6059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.783962 6059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:15.783975 6059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:15.789431 6059 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:35957
I20260927 06:59:15.789484 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:35957 every 8 connection(s)
I20260927 06:59:15.789788 6059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:15.793952 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6059
I20260927 06:59:15.794070 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:15.794235 3300 external_mini_cluster.cc:1428] Setting key ef862798b1f87ab1636e069ef7305b3a
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[5684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492355, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:15.798741 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.790492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:36655 (local address 127.3.57.62:41929)
0927 06:59:15.790544 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.790547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.790744 (+ 197us) server_negotiation.cc:407] Connection header received
0927 06:59:15.791174 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.791175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.791185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.791235 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.792141 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.792632 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.793541 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.793706 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.793753 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.796295 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:15.796296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:15.796297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:15.796303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:15.797692 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.798026 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.798027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.798028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.798079 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:15.798279 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:15.798279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:15.798280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:15.798373 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:15.798430 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:15.798624 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 06:59:15.798679 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 06:59:15.798921 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:15.790447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41929 (local address 127.3.57.3:36655)
0927 06:59:15.790618 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:15.790651 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:15.791096 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:15.791285 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:15.791288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:15.791887 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:15.792061 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.792065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.792716 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.792716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.793470 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:15.793472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:15.793789 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:15.793791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:15.793926 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:15.794570 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:15.794578 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:15.796085 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:15.797755 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.797760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.797762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.797961 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.798133 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:15.798133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:15.798134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:15.798225 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:15.798424 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:15.798427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:15.798523 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 06:59:15.798598 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:15.798702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260927 06:59:15.799355 6188 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41929
I20260927 06:59:15.799420 6188 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:15.799633 6188 heartbeater.cc:511] Master 127.3.57.62:41929 requested a full tablet report, sending...
I20260927 06:59:15.799988 5729 ts_manager.cc:194] Registered new tserver with Master: 6dcfbb40706e4928922a01d68c26770b (127.3.57.3:35957)
I20260927 06:59:15.800379 5729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:36655
I20260927 06:59:15.805677 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 06:59:15.816254 5769 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:15.815326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.3.57.62:41929)
0927 06:59:15.815406 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:15.815408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:15.815517 (+ 109us) server_negotiation.cc:407] Connection header received
0927 06:59:15.815653 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:15.815655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:15.815667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:15.815746 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:15.816170 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53300: BlockingRecv error: recv got EOF from 127.0.0.1:53300 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 06:59:15.817219 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5791
I20260927 06:59:15.821991 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5925
I20260927 06:59:15.826179 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6059
I20260927 06:59:15.830377 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 5697
2026-09-27T06:59:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1900 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[6200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[6200](info): set up 2 sockets
Sep 27 06:59:15 dist-test-slave-mk3h krb5kdc[6200](info): commencing operation
krb5kdc: starting...
W20260927 06:59:16.859349 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:16 dist-test-slave-mk3h krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:16.986198 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35125
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:17.064093 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:17.064345 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:17.064383 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:17.065517 6213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:17.065586 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:17.065611 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:17.065639 6213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:17.065668 6213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:17.067097 6213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:17.067360 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:17.067592 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:17.072288 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.072352 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:17.072352 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:17.072505 6213 server_base.cc:1034] running on GCE node
I20260927 06:59:17.072834 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:17.073143 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:17.074319 6213 hybrid_clock.cc:648] HybridClock initialized: now 1790492357074311 us; error 30 us; skew 500 ppm
Sep 27 06:59:17 dist-test-slave-mk3h krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492357, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:17.076629 6213 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:17.077049 6213 webserver.cc:460] Webserver started at http://127.3.57.62:40631/ using document root <none> and password file <none>
I20260927 06:59:17.077286 6213 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:17.077363 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:17.077543 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:17.078516 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "d71f12609f9c4b1fbb9dfce0354894bf"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-mk3h"
server_key: "f055c560ac1b5a902c0fab6a569e98fb"
server_key_iv: "cda5fc8f1c8e417569f6fbed6c1314ed"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.078871 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "d71f12609f9c4b1fbb9dfce0354894bf"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-mk3h"
server_key: "f055c560ac1b5a902c0fab6a569e98fb"
server_key_iv: "cda5fc8f1c8e417569f6fbed6c1314ed"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.080159 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:17.080816 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:17.080973 6213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:17.081065 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "d71f12609f9c4b1fbb9dfce0354894bf"
format_stamp: "Formatted at 2026-09-27 06:59:17 on dist-test-slave-mk3h"
server_key: "f055c560ac1b5a902c0fab6a569e98fb"
server_key_iv: "cda5fc8f1c8e417569f6fbed6c1314ed"
server_key_version: "encryptionkey@0"
I20260927 06:59:17.081142 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:17.101944 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:17.103667 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:17.103823 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:17.107499 6213 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35125
I20260927 06:59:17.107607 6280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35125 every 8 connection(s)
I20260927 06:59:17.107921 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:17.108700 6281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:17.110191 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6213
I20260927 06:59:17.110285 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:17.110477 3300 external_mini_cluster.cc:1428] Setting key da7fef4a863170ba062581407cb4b2d1
I20260927 06:59:17.111294 6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf: Bootstrap starting.
I20260927 06:59:17.111991 6281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:17.112327 6281 log.cc:824] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:17.113059 6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf: No bootstrap required, opened a new log
I20260927 06:59:17.115211 6281 raft_consensus.cc:348] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } }
I20260927 06:59:17.115392 6281 raft_consensus.cc:374] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:17.115422 6281 raft_consensus.cc:729] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71f12609f9c4b1fbb9dfce0354894bf, State: Initialized, Role: FOLLOWER
I20260927 06:59:17.115603 6281 consensus_queue.cc:260] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } }
I20260927 06:59:17.115710 6281 raft_consensus.cc:388] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:17.115741 6281 raft_consensus.cc:482] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:17.115794 6281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:17.116534 6281 raft_consensus.cc:504] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } }
I20260927 06:59:17.116648 6281 leader_election.cc:302] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d71f12609f9c4b1fbb9dfce0354894bf; no voters:
I20260927 06:59:17.116817 6281 leader_election.cc:288] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:17.116860 6286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:17.117031 6286 raft_consensus.cc:686] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [term 1 LEADER]: Becoming Leader. State: Replica: d71f12609f9c4b1fbb9dfce0354894bf, State: Running, Role: LEADER
I20260927 06:59:17.117106 6281 sys_catalog.cc:564] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 06:59:17 dist-test-slave-mk3h krb5kdc[6200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:17.117121 6286 consensus_queue.cc:237] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } }
I20260927 06:59:17.119071 6288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } } }
I20260927 06:59:17.119174 6288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:17.119364 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d71f12609f9c4b1fbb9dfce0354894bf. Latest consensus state: current_term: 1 leader_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d71f12609f9c4b1fbb9dfce0354894bf" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35125 } } }
I20260927 06:59:17.119419 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:17.119524 6301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:17.120014 6301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:17.121788 6301 catalog_manager.cc:1347] Generated new cluster ID: 742ba27d48334bd39d56b1e43c3c5629
I20260927 06:59:17.121816 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:17.110969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.3.57.62:35125)
0927 06:59:17.111125 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:17.111133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:17.111166 (+ 33us) server_negotiation.cc:407] Connection header received
0927 06:59:17.111659 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:17.111666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:17.111712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:17.111830 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:17.112658 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.113618 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.114679 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.115048 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.115122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:17.118033 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:17.118037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:17.118037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:17.118048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:17.120123 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:17.120533 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:17.120535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:17.120536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:17.120635 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:17.120842 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:17.120843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:17.120843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:17.121172 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:17.121333 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:17.121476 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 06:59:17.121614 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260927 06:59:17.121840 6301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:17.123015 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:17.134488 6301 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:17.135201 6301 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:17.138444 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:17.134219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.3.57.62:35125)
0927 06:59:17.134262 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:17.134264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:17.134381 (+ 117us) server_negotiation.cc:407] Connection header received
0927 06:59:17.134654 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:17.134655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:17.134665 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:17.134730 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:17.135124 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.135936 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.136926 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:17.137240 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:17.137315 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:17.137956 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:17.137957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:17.137957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:17.138155 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:17.138172 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:17.138179 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:17.138242 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:17.138325 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 06:59:17.138360 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 06:59:17.140151 6301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d71f12609f9c4b1fbb9dfce0354894bf: Generated new TSK 0
I20260927 06:59:17.140313 6301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:17.146569 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6213
2026-09-27T06:59:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1314 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:17 dist-test-slave-mk3h krb5kdc[6314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:17 dist-test-slave-mk3h krb5kdc[6314](info): set up 2 sockets
Sep 27 06:59:17 dist-test-slave-mk3h krb5kdc[6314](info): commencing operation
krb5kdc: starting...
W20260927 06:59:18.211359 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:18.323105 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42517
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:18.402267 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.402475 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.402506 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.403630 6327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:18.403690 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.403712 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.403745 6327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:18.403769 6327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:18.405236 6327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:18.405503 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.405717 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.410450 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.410451 6327 server_base.cc:1034] running on GCE node
W20260927 06:59:18.410467 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.410544 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.410951 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.411173 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.412331 6327 hybrid_clock.cc:648] HybridClock initialized: now 1790492358412323 us; error 38 us; skew 500 ppm
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:18.414595 6327 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:18.415000 6327 webserver.cc:460] Webserver started at http://127.3.57.62:41035/ using document root <none> and password file <none>
I20260927 06:59:18.415225 6327 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.415292 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.415446 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.416461 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "29426bdeb2904a1481bec4967d6c3b8f"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "3a447f2116de847930870d62a4304a71"
server_key_iv: "eea6171488ed2121e7eb6efbf397109a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.416806 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "29426bdeb2904a1481bec4967d6c3b8f"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "3a447f2116de847930870d62a4304a71"
server_key_iv: "eea6171488ed2121e7eb6efbf397109a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.418075 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:18.418763 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.418923 6327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:18.419019 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "29426bdeb2904a1481bec4967d6c3b8f"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "3a447f2116de847930870d62a4304a71"
server_key_iv: "eea6171488ed2121e7eb6efbf397109a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.419097 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.432521 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.434365 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.434525 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.438463 6327 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:42517
I20260927 06:59:18.438498 6394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:42517 every 8 connection(s)
I20260927 06:59:18.438827 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:18.439785 6395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:18.441900 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f: Bootstrap starting.
I20260927 06:59:18.442448 6395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:18.442724 6395 log.cc:824] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:18.443405 6395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f: No bootstrap required, opened a new log
I20260927 06:59:18.444995 6395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } }
I20260927 06:59:18.445122 6395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:18.445144 6395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29426bdeb2904a1481bec4967d6c3b8f, State: Initialized, Role: FOLLOWER
I20260927 06:59:18.445308 6395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } }
I20260927 06:59:18.445395 6395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:18.445447 6395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:18.445497 6395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:18.446267 6395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } }
I20260927 06:59:18.446379 6395 leader_election.cc:302] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29426bdeb2904a1481bec4967d6c3b8f; no voters:
I20260927 06:59:18.446560 6395 leader_election.cc:288] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:18.446637 6398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:18.446805 6398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [term 1 LEADER]: Becoming Leader. State: Replica: 29426bdeb2904a1481bec4967d6c3b8f, State: Running, Role: LEADER
I20260927 06:59:18.446837 6395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:18.446965 6398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } }
I20260927 06:59:18.447329 6400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29426bdeb2904a1481bec4967d6c3b8f. Latest consensus state: current_term: 1 leader_uuid: "29426bdeb2904a1481bec4967d6c3b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } } }
I20260927 06:59:18.447413 6400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:18.447686 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6327
I20260927 06:59:18.447672 6399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29426bdeb2904a1481bec4967d6c3b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29426bdeb2904a1481bec4967d6c3b8f" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42517 } } }
I20260927 06:59:18.447751 6399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:18.447788 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:18.447674 6406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:18.447979 3300 external_mini_cluster.cc:1428] Setting key 106e550b3cf4ae531aad27488e1a605b
I20260927 06:59:18.448352 6406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:18.449913 6406 catalog_manager.cc:1347] Generated new cluster ID: cff4ccc7b984486080f4778642a33a87
I20260927 06:59:18.449965 6406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:18.452819 6406 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:18.453269 6406 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:18.457073 6406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29426bdeb2904a1481bec4967d6c3b8f: Generated new TSK 0
I20260927 06:59:18.457261 6406 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:18.462141 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.448619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.3.57.62:42517)
0927 06:59:18.452430 (+ 3811us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.452437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.452460 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:18.452577 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.452584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.452640 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.452787 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.453508 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.454820 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.455919 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.456504 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.456599 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.458982 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.458986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.458986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:18.459011 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.460666 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.461047 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.461049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.461050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.461142 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.461355 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.461355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.461356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.461558 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.461695 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.461810 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.461967 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3772,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:18.488543 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:18.566267 6421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.566483 6421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.566520 6421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.567662 6421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.567724 6421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.567770 6421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:18.569166 6421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:18.569419 6421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.569634 6421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.574554 6427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.574630 6426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.574617 6429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.574618 6421 server_base.cc:1034] running on GCE node
I20260927 06:59:18.574918 6421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.575104 6421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.576231 6421 hybrid_clock.cc:648] HybridClock initialized: now 1790492358576211 us; error 36 us; skew 500 ppm
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:18.578616 6421 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:18.578967 6421 webserver.cc:460] Webserver started at http://127.3.57.1:42245/ using document root <none> and password file <none>
I20260927 06:59:18.579133 6421 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.579172 6421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.579255 6421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.580287 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "4905024c539544578c634fc09167d295"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "203a78df8acf839bde27ebf88fc8824a"
server_key_iv: "4dc8bca9240cd1c48b47735dfc95353f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.580574 6421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "4905024c539544578c634fc09167d295"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "203a78df8acf839bde27ebf88fc8824a"
server_key_iv: "4dc8bca9240cd1c48b47735dfc95353f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.581802 6421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:18.582594 6436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.582755 6421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:18.582830 6421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "4905024c539544578c634fc09167d295"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "203a78df8acf839bde27ebf88fc8824a"
server_key_iv: "4dc8bca9240cd1c48b47735dfc95353f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.582881 6421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.592691 6421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.594310 6421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.594409 6421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.594566 6421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:18.594797 6421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:18.594825 6421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.594846 6421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:18.594858 6421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.600252 6421 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:35461
I20260927 06:59:18.600327 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:35461 every 8 connection(s)
I20260927 06:59:18.600608 6421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:18.602905 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6421
I20260927 06:59:18.602996 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:18.603184 3300 external_mini_cluster.cc:1428] Setting key 0a1052f5a0e5a9b1f40dc1d2a5e2a860
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:18.610694 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.601283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:45133 (local address 127.3.57.62:42517)
0927 06:59:18.601327 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.601329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.601515 (+ 186us) server_negotiation.cc:407] Connection header received
0927 06:59:18.601975 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.601976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.601985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.602039 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.602996 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.603742 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.605218 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.605556 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.605629 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.607913 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.607915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.607916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:18.607924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.609333 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.609735 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.609736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.609737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.609799 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.610047 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.610048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.610049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.610162 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.610240 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.610511 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.610606 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 06:59:18.610826 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.601249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42517 (local address 127.3.57.1:45133)
0927 06:59:18.601403 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.601443 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.601895 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.602088 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.602090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.602621 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:18.602881 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.602888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.603810 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.603811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.605129 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.605132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.605639 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.605640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.605735 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.606213 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:18.606220 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:18.607815 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:18.609439 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.609446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.609448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.609659 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.609865 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.609866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.609867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.609978 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.610232 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:18.610238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:18.610371 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.610472 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.610619 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 06:59:18.611335 6550 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42517
I20260927 06:59:18.611439 6550 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
I20260927 06:59:18.611651 6550 heartbeater.cc:511] Master 127.3.57.62:42517 requested a full tablet report, sending...
I20260927 06:59:18.612154 6359 ts_manager.cc:194] Registered new tserver with Master: 4905024c539544578c634fc09167d295 (127.3.57.1:35461)
I20260927 06:59:18.612890 6359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:45133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:18.628090 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:18.707238 6555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.707417 6555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.707440 6555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.708551 6555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.708585 6555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.708611 6555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:18.709875 6555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:18.710041 6555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.710206 6555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.714764 6561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.714754 6560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.714744 6563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.715000 6555 server_base.cc:1034] running on GCE node
I20260927 06:59:18.715138 6555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.715305 6555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.716432 6555 hybrid_clock.cc:648] HybridClock initialized: now 1790492358716421 us; error 26 us; skew 500 ppm
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:18.718801 6555 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:18.719166 6555 webserver.cc:460] Webserver started at http://127.3.57.2:45269/ using document root <none> and password file <none>
I20260927 06:59:18.719336 6555 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.719372 6555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.719457 6555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.720455 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "31258e77298542728396fe20fd28d405"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "957b56b54a45afa024e64b5988d790e9"
server_key_iv: "2bc05176a0b9cd9bbe9b1c2de440679a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.720767 6555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "31258e77298542728396fe20fd28d405"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "957b56b54a45afa024e64b5988d790e9"
server_key_iv: "2bc05176a0b9cd9bbe9b1c2de440679a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.722057 6555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 06:59:18.722833 6570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.723037 6555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:18.723107 6555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "31258e77298542728396fe20fd28d405"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "957b56b54a45afa024e64b5988d790e9"
server_key_iv: "2bc05176a0b9cd9bbe9b1c2de440679a"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.723197 6555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.747792 6555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.749706 6555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.749859 6555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.750077 6555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:18.750386 6555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:18.750419 6555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.750442 6555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:18.750490 6555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.756201 6555 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:43779
I20260927 06:59:18.756258 6683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:43779 every 8 connection(s)
I20260927 06:59:18.756548 6555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:18.762317 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6555
I20260927 06:59:18.762457 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:18.762653 3300 external_mini_cluster.cc:1428] Setting key bf517c9f606f858a0ecc6173a2fdbac3
I20260927 06:59:18.765179 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.757223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:48307 (local address 127.3.57.62:42517)
0927 06:59:18.757263 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.757265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.757470 (+ 205us) server_negotiation.cc:407] Connection header received
0927 06:59:18.757883 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.757884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.757893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.757941 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.758749 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.759212 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.760108 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.760281 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.760330 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.762171 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.762172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.762173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:18.762179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.763722 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.764196 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.764197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.764197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.764244 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.764542 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.764543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.764543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.764669 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.764728 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.765056 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.765119 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 06:59:18.765457 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.757213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42517 (local address 127.3.57.2:48307)
0927 06:59:18.757354 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.757386 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.757806 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.757990 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.757993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.758496 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:18.758675 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.758679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.759285 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.759286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.760037 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.760038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.760348 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.760349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.760437 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.760799 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:18.760806 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:18.762091 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:18.763793 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.763800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.763802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.764120 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.764302 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.764302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.764303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.764458 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.764737 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:18.764743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:18.764907 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.765037 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.765215 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260927 06:59:18.766042 6684 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42517
I20260927 06:59:18.766157 6684 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:18.766335 6684 heartbeater.cc:511] Master 127.3.57.62:42517 requested a full tablet report, sending...
I20260927 06:59:18.766726 6359 ts_manager.cc:194] Registered new tserver with Master: 31258e77298542728396fe20fd28d405 (127.3.57.2:43779)
I20260927 06:59:18.767104 6359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:48307
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:18.786828 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:18.864694 6689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:18.864919 6689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:18.864957 6689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:18.866086 6689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:18.866151 6689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:18.866195 6689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:18.867612 6689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:18.867864 6689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:18.868076 6689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:18.872860 6695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.872911 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:18.873010 6697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:18.872983 6689 server_base.cc:1034] running on GCE node
I20260927 06:59:18.873242 6689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:18.873449 6689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:18.874585 6689 hybrid_clock.cc:648] HybridClock initialized: now 1790492358874570 us; error 29 us; skew 500 ppm
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:18.877117 6689 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:18.877530 6689 webserver.cc:460] Webserver started at http://127.3.57.3:33169/ using document root <none> and password file <none>
I20260927 06:59:18.877744 6689 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:18.877810 6689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:18.877960 6689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:18.879089 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "4f5aaa8e47a04ab8beb97b905c0a7bba"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "186bd6a0044f529225361bf29cddd4c3"
server_key_iv: "a2d75122385e0b757b9d1966e0e2470f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.879472 6689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "4f5aaa8e47a04ab8beb97b905c0a7bba"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "186bd6a0044f529225361bf29cddd4c3"
server_key_iv: "a2d75122385e0b757b9d1966e0e2470f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.880846 6689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 06:59:18.881560 6704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.881722 6689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:18.881822 6689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "4f5aaa8e47a04ab8beb97b905c0a7bba"
format_stamp: "Formatted at 2026-09-27 06:59:18 on dist-test-slave-mk3h"
server_key: "186bd6a0044f529225361bf29cddd4c3"
server_key_iv: "a2d75122385e0b757b9d1966e0e2470f"
server_key_version: "encryptionkey@0"
I20260927 06:59:18.881898 6689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:18.891667 6689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:18.893883 6689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:18.894049 6689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:18.894279 6689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:18.894575 6689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:18.894630 6689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.894665 6689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:18.894691 6689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:18.900367 6689 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33655
I20260927 06:59:18.900408 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33655 every 8 connection(s)
I20260927 06:59:18.900707 6689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:18.900986 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6689
I20260927 06:59:18.901088 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:18.901289 3300 external_mini_cluster.cc:1428] Setting key 3241fc8a2e6578b80f1c31d8b6f7fee9
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:18.909734 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.901427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:58487 (local address 127.3.57.62:42517)
0927 06:59:18.901494 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.901496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.901740 (+ 244us) server_negotiation.cc:407] Connection header received
0927 06:59:18.902377 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.902378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.902388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.902429 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.903289 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.903830 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.904813 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.905162 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.905287 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.907239 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.907240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.907241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:18.907249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.908581 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.908936 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.908937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.908937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.908990 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.909210 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.909211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.909211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.909322 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.909385 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.909610 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.909672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 06:59:18.909896 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.901387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42517 (local address 127.3.57.3:58487)
0927 06:59:18.901549 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:18.901652 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:18.902280 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:18.902495 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:18.902498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:18.903040 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:18.903217 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.903221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.903926 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.903927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.904733 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:18.904735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.905302 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.905303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:18.905399 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.905850 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:18.905857 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:18.907143 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:18.908656 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.908661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.908662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.908868 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.909049 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:18.909050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:18.909050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:18.909139 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:18.909390 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:18.909393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:18.909503 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 06:59:18.909578 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:18.909685 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260927 06:59:18.910356 6818 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42517
I20260927 06:59:18.910429 6818 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:18.910604 6818 heartbeater.cc:511] Master 127.3.57.62:42517 requested a full tablet report, sending...
I20260927 06:59:18.910987 6359 ts_manager.cc:194] Registered new tserver with Master: 4f5aaa8e47a04ab8beb97b905c0a7bba (127.3.57.3:33655)
I20260927 06:59:18.911355 6359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:58487
I20260927 06:59:18.912954 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 06:59:18 dist-test-slave-mk3h krb5kdc[6314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492358, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:18.934482 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:18.927947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.3.57.62:42517)
0927 06:59:18.928022 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:18.928024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:18.928113 (+ 89us) server_negotiation.cc:407] Connection header received
0927 06:59:18.928151 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:18.928151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:18.928160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:18.928206 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:18.928898 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.929352 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.930069 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:18.930262 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:18.930308 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:18.932398 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:18.932399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:18.932399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:18.932405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:18.933526 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.933843 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.933844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.933844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.933888 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:18.934052 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:18.934052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:18.934053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:18.934173 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:18.934243 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:18.934323 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 06:59:18.934388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 06:59:18.934713 6359 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49940
I20260927 06:59:18.935091 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6421
I20260927 06:59:18.939610 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6555
I20260927 06:59:18.943660 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6689
I20260927 06:59:18.947655 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6327
2026-09-27T06:59:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1803 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:19 dist-test-slave-mk3h krb5kdc[6830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:19 dist-test-slave-mk3h krb5kdc[6830](info): set up 2 sockets
Sep 27 06:59:19 dist-test-slave-mk3h krb5kdc[6830](info): commencing operation
krb5kdc: starting...
W20260927 06:59:19.983403 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:20 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:20.094390 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35657
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:20.175693 6843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:20.175889 6843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:20.175910 6843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:20.177515 6843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:20.177551 6843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:20.177564 6843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:20.177577 6843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:20.177589 6843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:20.179174 6843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:20.179359 6843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:20.179518 6843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:20.184175 6849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:20.184190 6848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:20.184294 6851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:20.184428 6843 server_base.cc:1034] running on GCE node
I20260927 06:59:20.184654 6843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:20.184954 6843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:20.186146 6843 hybrid_clock.cc:648] HybridClock initialized: now 1790492360186120 us; error 36 us; skew 500 ppm
Sep 27 06:59:20 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:20.188961 6843 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:20.189459 6843 webserver.cc:460] Webserver started at http://127.3.57.62:38997/ using document root <none> and password file <none>
I20260927 06:59:20.189689 6843 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:20.189733 6843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:20.189852 6843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:20.190825 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "b09fde86ef454a8d9401ae70783fa609"
format_stamp: "Formatted at 2026-09-27 06:59:20 on dist-test-slave-mk3h"
server_key: "5444628fac773656ea67cbf910d13783"
server_key_iv: "4aff25239141dae2e37e0bf27900cdb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:20.191182 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "b09fde86ef454a8d9401ae70783fa609"
format_stamp: "Formatted at 2026-09-27 06:59:20 on dist-test-slave-mk3h"
server_key: "5444628fac773656ea67cbf910d13783"
server_key_iv: "4aff25239141dae2e37e0bf27900cdb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:20.192663 6843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:20.193370 6858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:20.193559 6843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:20.193651 6843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "b09fde86ef454a8d9401ae70783fa609"
format_stamp: "Formatted at 2026-09-27 06:59:20 on dist-test-slave-mk3h"
server_key: "5444628fac773656ea67cbf910d13783"
server_key_iv: "4aff25239141dae2e37e0bf27900cdb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:20.193701 6843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:20.205693 6843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:20.207326 6843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:20.207430 6843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:20.211031 6843 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35657
I20260927 06:59:20.211076 6910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35657 every 8 connection(s)
I20260927 06:59:20.211526 6843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:20.212354 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:20.215860 6911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: Bootstrap starting.
I20260927 06:59:20.216449 6911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:20.216687 6911 log.cc:824] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:20.217278 6911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: No bootstrap required, opened a new log
I20260927 06:59:20.218544 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6843
I20260927 06:59:20.218619 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:20.218775 3300 external_mini_cluster.cc:1428] Setting key 7e6e48a5865d1c7cc04de1d33afb1da9
I20260927 06:59:20.219094 6911 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } }
I20260927 06:59:20.219301 6911 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:20.219365 6911 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09fde86ef454a8d9401ae70783fa609, State: Initialized, Role: FOLLOWER
I20260927 06:59:20.219540 6911 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } }
I20260927 06:59:20.219666 6911 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:20.219702 6911 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:20.219758 6911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 06:59:20 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:20.228014 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:20.219268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.3.57.62:35657)
0927 06:59:20.219908 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:20.219915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:20.219939 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:20.220036 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:20.220041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:20.220081 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:20.220216 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:20.220844 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:20.221707 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:20.222440 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:20.222837 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:20.222945 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:20.225144 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:20.225147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:20.225147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:20.225158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:20.226662 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:20.226995 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:20.226998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:20.226998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:20.227108 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:20.227271 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:20.227272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:20.227272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:20.227490 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:20.227641 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:20.227732 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 06:59:20.227865 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":561,"threads_started":1}
I20260927 06:59:20.284405 6911 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } }
I20260927 06:59:20.284617 6911 leader_election.cc:302] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09fde86ef454a8d9401ae70783fa609; no voters:
I20260927 06:59:20.284824 6911 leader_election.cc:288] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:20.284906 6916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:20.285104 6911 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:20.285107 6916 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [term 1 LEADER]: Becoming Leader. State: Replica: b09fde86ef454a8d9401ae70783fa609, State: Running, Role: LEADER
I20260927 06:59:20.285362 6916 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } }
I20260927 06:59:20.285759 6917 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09fde86ef454a8d9401ae70783fa609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } } }
I20260927 06:59:20.285838 6917 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:20.286077 6918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09fde86ef454a8d9401ae70783fa609. Latest consensus state: current_term: 1 leader_uuid: "b09fde86ef454a8d9401ae70783fa609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09fde86ef454a8d9401ae70783fa609" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35657 } } }
I20260927 06:59:20.286444 6918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609 [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:20.287147 6932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:20.287204 6932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:20.287246 6924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:20.287396 6924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:20.289000 6924 catalog_manager.cc:1347] Generated new cluster ID: 2b513ac7e4874af090146cc15a28b534
I20260927 06:59:20.289053 6924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:20.297708 6924 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:20.298418 6924 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:20.302901 6924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09fde86ef454a8d9401ae70783fa609: Generated new TSK 0
I20260927 06:59:20.303064 6924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
W20260927 06:59:21.465849 3300 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.57.1: real 1.157s user 0.000s sys 0.003s
I20260927 06:59:21.466238 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:21.547771 6937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:21.547966 6937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:21.547987 6937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:21.549063 6937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:21.549098 6937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:21.549122 6937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:21.550585 6937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:21.550761 6937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:21.550923 6937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:21.555778 6942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:21.555902 6937 server_base.cc:1034] running on GCE node
W20260927 06:59:21.555756 6945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:21.556670 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:21.556912 6937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:21.557106 6937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:21.558248 6937 hybrid_clock.cc:648] HybridClock initialized: now 1790492361558218 us; error 46 us; skew 500 ppm
Sep 27 06:59:21 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492361, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:21.560698 6937 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:21.561048 6937 webserver.cc:460] Webserver started at http://127.3.57.1:41059/ using document root <none> and password file <none>
I20260927 06:59:21.561275 6937 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:21.561339 6937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:21.561512 6937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:21.562496 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "7e78895383f44e2a9ab65f16418d9d2f"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-mk3h"
server_key: "2f117c1ce703188e6c608e858d0c2879"
server_key_iv: "6f7c4bc37de47e292a07389aac3ca04c"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.562898 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "7e78895383f44e2a9ab65f16418d9d2f"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-mk3h"
server_key: "2f117c1ce703188e6c608e858d0c2879"
server_key_iv: "6f7c4bc37de47e292a07389aac3ca04c"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.564306 6937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:21.565152 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:21.565348 6937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:21.565438 6937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "7e78895383f44e2a9ab65f16418d9d2f"
format_stamp: "Formatted at 2026-09-27 06:59:21 on dist-test-slave-mk3h"
server_key: "2f117c1ce703188e6c608e858d0c2879"
server_key_iv: "6f7c4bc37de47e292a07389aac3ca04c"
server_key_version: "encryptionkey@0"
I20260927 06:59:21.565531 6937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:21.580090 6937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:21.582612 6937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:21.582718 6937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:21.582899 6937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:21.583134 6937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:21.583163 6937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:21.583184 6937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:21.583196 6937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:21.589458 6937 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:34625
I20260927 06:59:21.589496 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:34625 every 8 connection(s)
I20260927 06:59:21.589804 6937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:21.590973 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 6937
I20260927 06:59:21.591084 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:21.591274 3300 external_mini_cluster.cc:1428] Setting key 053b5636cd2932a4464aa4afa7260253
Sep 27 06:59:21 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492361, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
I20260927 06:59:21.600658 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:21.590558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:44219 (local address 127.3.57.62:35657)
0927 06:59:21.590690 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:21.590692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:21.590843 (+ 151us) server_negotiation.cc:407] Connection header received
0927 06:59:21.592033 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:21.592034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:21.592047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:21.592106 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:21.592708 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.593500 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.594776 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.595112 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.595191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:21.597802 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:21.597803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:21.597803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:21.597809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:21.599245 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.599755 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.599755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.599756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.599810 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:21.600110 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:21.600111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:21.600111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:21.600213 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:21.600263 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:21.600529 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 06:59:21.600590 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260927 06:59:21.600807 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:21.590528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35657 (local address 127.3.57.1:44219)
0927 06:59:21.590698 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:21.590761 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:21.591938 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:21.592175 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:21.592179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:21.592427 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:21.592614 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:21.592618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.593597 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.593598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:21.594684 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:21.594686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:21.595218 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:21.595219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:21.595325 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:21.595910 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:21.595918 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:21.597706 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:21.599334 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:21.599341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:21.599344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:21.599680 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:21.599886 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:21.599887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:21.599888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:21.600050 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:21.600285 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:21.600291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:21.600402 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 06:59:21.600488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 06:59:21.600602 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260927 06:59:21.601318 7066 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35657
I20260927 06:59:21.601399 7066 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:21.601534 7066 heartbeater.cc:511] Master 127.3.57.62:35657 requested a full tablet report, sending...
I20260927 06:59:21.601979 6875 ts_manager.cc:194] Registered new tserver with Master: 7e78895383f44e2a9ab65f16418d9d2f (127.3.57.1:34625)
I20260927 06:59:21.602576 6875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:44219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
W20260927 06:59:22.584142 3300 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.57.2: real 0.992s user 0.001s sys 0.002s
I20260927 06:59:22.584530 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:59:22.603682 7066 heartbeater.cc:503] Master 127.3.57.62:35657 was elected leader, sending a full tablet report...
W20260927 06:59:22.665393 7072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:22.665628 7072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:22.665666 7072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:22.666885 7072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:22.666946 7072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:22.666992 7072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:22.668562 7072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:22.668825 7072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:22.669049 7072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:22.675040 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:22.675058 7080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:22.675058 7077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:22.675180 7072 server_base.cc:1034] running on GCE node
I20260927 06:59:22.675480 7072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:22.675819 7072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:22.677003 7072 hybrid_clock.cc:648] HybridClock initialized: now 1790492362676986 us; error 28 us; skew 500 ppm
Sep 27 06:59:22 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:22.679656 7072 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:22.680121 7072 webserver.cc:460] Webserver started at http://127.3.57.2:40387/ using document root <none> and password file <none>
I20260927 06:59:22.680377 7072 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:22.680449 7072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:22.680601 7072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:22.681638 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "64e78595a57148d8b53c0db1f21aaf19"
format_stamp: "Formatted at 2026-09-27 06:59:22 on dist-test-slave-mk3h"
server_key: "7188d84b1e1396d57743eeb655e97b23"
server_key_iv: "4d128d1e097e150aa3f2a05832ba6674"
server_key_version: "encryptionkey@0"
I20260927 06:59:22.682001 7072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "64e78595a57148d8b53c0db1f21aaf19"
format_stamp: "Formatted at 2026-09-27 06:59:22 on dist-test-slave-mk3h"
server_key: "7188d84b1e1396d57743eeb655e97b23"
server_key_iv: "4d128d1e097e150aa3f2a05832ba6674"
server_key_version: "encryptionkey@0"
I20260927 06:59:22.683786 7072 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:22.684643 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:22.684857 7072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:22.684942 7072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "64e78595a57148d8b53c0db1f21aaf19"
format_stamp: "Formatted at 2026-09-27 06:59:22 on dist-test-slave-mk3h"
server_key: "7188d84b1e1396d57743eeb655e97b23"
server_key_iv: "4d128d1e097e150aa3f2a05832ba6674"
server_key_version: "encryptionkey@0"
I20260927 06:59:22.684996 7072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:22.725239 7072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:22.727466 7072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:22.727650 7072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:22.727864 7072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:22.728106 7072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:22.728133 7072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:22.728156 7072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:22.728168 7072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:22.734028 7072 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46205
I20260927 06:59:22.734102 7200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46205 every 8 connection(s)
I20260927 06:59:22.734441 7072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:22.740214 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7072
I20260927 06:59:22.740302 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:22.740499 3300 external_mini_cluster.cc:1428] Setting key 5ba2f2613439bcff5d69c49c7fc35109
Sep 27 06:59:22 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:22.743891 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:22.735131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35657 (local address 127.3.57.2:40729)
0927 06:59:22.735304 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:22.735346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:22.736339 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:22.736602 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:22.736607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:22.736853 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:22.737027 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:22.737031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:22.737725 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:22.737726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:22.738509 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:22.738511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:22.738918 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:22.738919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:22.739041 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:22.739446 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:22.739454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:22.740895 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:22.742523 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:22.742529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:22.742531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:22.742746 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:22.742976 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:22.742977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:22.742977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:22.743081 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:22.743338 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:22.743343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:22.743480 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 06:59:22.743599 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 06:59:22.743735 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260927 06:59:22.744258 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:22.735156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:40729 (local address 127.3.57.62:35657)
0927 06:59:22.735297 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:22.735300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:22.735434 (+ 134us) server_negotiation.cc:407] Connection header received
0927 06:59:22.736446 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:22.736447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:22.736464 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:22.736523 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:22.737119 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:22.737635 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:22.738609 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:22.738828 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:22.738909 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:22.740990 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:22.740992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:22.740992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:22.741000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:22.742414 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:22.742827 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:22.742829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:22.742829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:22.742890 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:22.743152 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:22.743153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:22.743154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:22.743268 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:22.743343 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:22.744079 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 06:59:22.744160 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260927 06:59:22.744793 7201 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35657
I20260927 06:59:22.744890 7201 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:22.745082 7201 heartbeater.cc:511] Master 127.3.57.62:35657 requested a full tablet report, sending...
I20260927 06:59:22.745441 6875 ts_manager.cc:194] Registered new tserver with Master: 64e78595a57148d8b53c0db1f21aaf19 (127.3.57.2:46205)
I20260927 06:59:22.745810 6875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:40729
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
I20260927 06:59:23.746791 7201 heartbeater.cc:503] Master 127.3.57.62:35657 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
W20260927 06:59:23.873713 3300 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.57.3: real 1.133s user 0.000s sys 0.003s
I20260927 06:59:23.874065 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:23.954730 7207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:23.954934 7207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:23.954958 7207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:23.956103 7207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:23.956163 7207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:23.956207 7207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:23.957671 7207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:23.957924 7207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:23.958143 7207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:23.963069 7215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:23.963073 7213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:23.963073 7212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:23.963102 7207 server_base.cc:1034] running on GCE node
I20260927 06:59:23.963510 7207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:23.963742 7207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:23.964877 7207 hybrid_clock.cc:648] HybridClock initialized: now 1790492363964865 us; error 28 us; skew 500 ppm
Sep 27 06:59:23 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:23.967391 7207 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:23.967845 7207 webserver.cc:460] Webserver started at http://127.3.57.3:43429/ using document root <none> and password file <none>
I20260927 06:59:23.968113 7207 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:23.968168 7207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:23.968320 7207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:23.969383 7207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "446029b557724e838a7336d008bba019"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-mk3h"
server_key: "a49a1f7f660be6648c37643a29a21321"
server_key_iv: "7b66ff1113a23e82c8f50bbb3bb2186d"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.969822 7207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "446029b557724e838a7336d008bba019"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-mk3h"
server_key: "a49a1f7f660be6648c37643a29a21321"
server_key_iv: "7b66ff1113a23e82c8f50bbb3bb2186d"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.971156 7207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:23.971956 7222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:23.972153 7207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:23.972255 7207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "446029b557724e838a7336d008bba019"
format_stamp: "Formatted at 2026-09-27 06:59:23 on dist-test-slave-mk3h"
server_key: "a49a1f7f660be6648c37643a29a21321"
server_key_iv: "7b66ff1113a23e82c8f50bbb3bb2186d"
server_key_version: "encryptionkey@0"
I20260927 06:59:23.972435 7207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:23.986876 7207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:23.989599 7207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:23.989718 7207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:23.989898 7207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:23.990195 7207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:23.990257 7207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:23.990309 7207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:23.990337 7207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:23.996069 7207 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33631
I20260927 06:59:23.996109 7335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33631 every 8 connection(s)
I20260927 06:59:23.996472 7207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:23.998588 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7207
I20260927 06:59:23.998669 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:23.998863 3300 external_mini_cluster.cc:1428] Setting key 8eb035554c21cc4ea61d4e100388390b
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:24.009513 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:23.997173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:57261 (local address 127.3.57.62:35657)
0927 06:59:23.997284 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:23.997286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:23.997414 (+ 128us) server_negotiation.cc:407] Connection header received
0927 06:59:23.998351 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:23.998352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:23.998363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:23.998475 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:23.999170 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.999917 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.000869 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.001054 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.001102 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:24.006910 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:24.006911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:24.006912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:24.006918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:24.008333 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.008723 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.008724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.008724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.008774 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.008975 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.008975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.008976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.009078 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:24.009127 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:24.009367 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 06:59:24.009441 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260927 06:59:24.009654 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:23.997145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35657 (local address 127.3.57.3:57261)
0927 06:59:23.997299 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:23.997339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:23.998256 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:23.998521 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:23.998524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:23.998832 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:23.999063 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:23.999068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:23.999994 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:23.999995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:24.000797 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:24.000799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.001121 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.001122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:24.001219 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:24.005409 (+ 4190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:24.005418 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:24.006804 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:24.008408 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:24.008414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:24.008415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:24.008661 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:24.008830 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:24.008831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:24.008831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:24.008926 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:24.009128 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:24.009132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:24.009242 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 06:59:24.009347 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 06:59:24.009465 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260927 06:59:24.010157 7336 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35657
I20260927 06:59:24.010274 7336 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:24.010474 7336 heartbeater.cc:511] Master 127.3.57.62:35657 requested a full tablet report, sending...
I20260927 06:59:24.010824 6875 ts_manager.cc:194] Registered new tserver with Master: 446029b557724e838a7336d008bba019 (127.3.57.3:33631)
I20260927 06:59:24.011195 6875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:57261
I20260927 06:59:24.021293 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[6830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:24.061216 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:24.053199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.3.57.1:34625)
0927 06:59:24.053324 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:24.053330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:24.053348 (+ 18us) server_negotiation.cc:407] Connection header received
0927 06:59:24.053377 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:24.053379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:24.053399 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:24.053462 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:24.054261 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.054940 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.056210 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.056516 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.056570 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:24.058878 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:24.058885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:24.058885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:24.058891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:24.060334 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.060638 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.060640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.060641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.060693 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.060833 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.060834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.060834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.060939 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:24.061049 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:24.061085 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 06:59:24.061140 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[6830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:24.068488 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:24.062410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.3.57.62:35657)
0927 06:59:24.062451 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:24.062457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:24.062465 (+ 8us) server_negotiation.cc:407] Connection header received
0927 06:59:24.062517 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:24.062518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:24.062526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:24.062571 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:24.062991 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.063467 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.064188 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:24.064372 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:24.064433 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:24.066418 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:24.066420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:24.066420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:24.066426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:24.067624 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.067913 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.067914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.067914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.067959 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:24.068114 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:24.068114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:24.068114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:24.068235 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:24.068300 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:24.068364 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 06:59:24.068432 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 06:59:24.068725 6875 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45924
I20260927 06:59:24.069051 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6937
I20260927 06:59:24.073899 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7072
I20260927 06:59:24.078392 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7207
I20260927 06:59:24.082314 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 6843
2026-09-27T06:59:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5134 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[7349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[7349](info): set up 2 sockets
Sep 27 06:59:24 dist-test-slave-mk3h krb5kdc[7349](info): commencing operation
krb5kdc: starting...
W20260927 06:59:25.667519 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.565s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:25 dist-test-slave-mk3h krb5kdc[7349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:25.782346 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:46865
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:46865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:25.859710 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:25.859928 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:25.859964 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:25.861097 7362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:25.861161 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:25.861186 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:25.861215 7362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:25.861239 7362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:25.862686 7362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:46865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:25.862949 7362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:25.863153 7362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:25.867799 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:25.867781 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:25.867774 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:25.867835 7362 server_base.cc:1034] running on GCE node
I20260927 06:59:25.868201 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:25.868476 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:25.869632 7362 hybrid_clock.cc:648] HybridClock initialized: now 1790492365869616 us; error 29 us; skew 500 ppm
Sep 27 06:59:25 dist-test-slave-mk3h krb5kdc[7349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492365, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:25.871981 7362 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:25.872314 7362 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:25.872370 7362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:25.872473 7362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:25.873409 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "d4bf5018f091441194d37024ef1edeef"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-mk3h"
server_key: "44cea37711a7228d03539f0f8057ae53"
server_key_iv: "0ed1d36a870390d0935061e566fe5058"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.873703 7362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "d4bf5018f091441194d37024ef1edeef"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-mk3h"
server_key: "44cea37711a7228d03539f0f8057ae53"
server_key_iv: "0ed1d36a870390d0935061e566fe5058"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.874837 7362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 06:59:25.875440 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:25.875630 7362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:25.875718 7362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "d4bf5018f091441194d37024ef1edeef"
format_stamp: "Formatted at 2026-09-27 06:59:25 on dist-test-slave-mk3h"
server_key: "44cea37711a7228d03539f0f8057ae53"
server_key_iv: "0ed1d36a870390d0935061e566fe5058"
server_key_version: "encryptionkey@0"
I20260927 06:59:25.875787 7362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:25.883246 7362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:25.885282 7362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:25.885430 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:25.889101 7362 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:46865
I20260927 06:59:25.889137 7428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:46865 every 8 connection(s)
I20260927 06:59:25.889390 7362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:25.890379 7429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:25.892838 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: Bootstrap starting.
I20260927 06:59:25.893350 7429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:25.893597 7429 log.cc:824] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:25.894166 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: No bootstrap required, opened a new log
I20260927 06:59:25.895709 7429 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } }
I20260927 06:59:25.895836 7429 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:25.895859 7429 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bf5018f091441194d37024ef1edeef, State: Initialized, Role: FOLLOWER
I20260927 06:59:25.895995 7429 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } }
I20260927 06:59:25.896076 7429 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:25.896100 7429 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:25.896165 7429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:25.896178 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7362
I20260927 06:59:25.896292 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:25.896476 3300 external_mini_cluster.cc:1428] Setting key 6ee4895d3b8d08a72979b525aa7d8479
I20260927 06:59:25.897171 7429 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } }
I20260927 06:59:25.897286 7429 leader_election.cc:302] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4bf5018f091441194d37024ef1edeef; no voters:
I20260927 06:59:25.897442 7429 leader_election.cc:288] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:25.897476 7434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:25.897574 7434 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [term 1 LEADER]: Becoming Leader. State: Replica: d4bf5018f091441194d37024ef1edeef, State: Running, Role: LEADER
I20260927 06:59:25.897652 7434 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } }
I20260927 06:59:25.897686 7429 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:25.897972 7434 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4bf5018f091441194d37024ef1edeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } } }
I20260927 06:59:25.898134 7434 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:25.898314 7434 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4bf5018f091441194d37024ef1edeef. Latest consensus state: current_term: 1 leader_uuid: "d4bf5018f091441194d37024ef1edeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4bf5018f091441194d37024ef1edeef" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46865 } } }
I20260927 06:59:25.898417 7434 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:25.898772 7442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:25.899406 7442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:25.901036 7442 catalog_manager.cc:1347] Generated new cluster ID: b4cbc03514d64e079f317e410358ba09
I20260927 06:59:25.901098 7442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 06:59:25 dist-test-slave-mk3h krb5kdc[7349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:25.905061 7442 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:25.905869 7442 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:25.908213 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:25.896935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.3.57.62:46865)
0927 06:59:25.897109 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:25.897117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:25.897144 (+ 27us) server_negotiation.cc:407] Connection header received
0927 06:59:25.897229 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:25.897234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:25.897290 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:25.897417 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:25.898290 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:25.899676 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:25.900684 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:25.901271 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:25.901423 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:25.904118 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:25.904121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:25.904121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:25.904132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:25.906571 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:25.906961 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:25.906964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:25.906964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:25.907064 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:25.907304 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:25.907305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:25.907305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:25.907558 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:25.907751 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:25.907872 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 06:59:25.908008 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260927 06:59:25.909546 7442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: Generated new TSK 0
I20260927 06:59:25.909704 7442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:25.938046 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:26.016103 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.016299 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.016320 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.017423 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.017462 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.017508 7455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:26.018957 7455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.019184 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.019411 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:59:26.024896 7455 server_base.cc:1034] running on GCE node
W20260927 06:59:26.024866 7461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.024899 7460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.024899 7463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.025231 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.025480 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.026619 7455 hybrid_clock.cc:648] HybridClock initialized: now 1790492366026609 us; error 30 us; skew 500 ppm
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:26.028995 7455 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:26.029345 7455 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.029419 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.029549 7455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.030531 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "959d59e18591178311886849fb6e0c4f"
server_key_iv: "7e65798bed7b881fe0fb0f5ebe33f23e"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.030877 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "959d59e18591178311886849fb6e0c4f"
server_key_iv: "7e65798bed7b881fe0fb0f5ebe33f23e"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.032192 7455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:26.032871 7469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.033030 7455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:26.033124 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "959d59e18591178311886849fb6e0c4f"
server_key_iv: "7e65798bed7b881fe0fb0f5ebe33f23e"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.033192 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.049162 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.050966 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.051118 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.051339 7455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:26.051687 7455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:26.051734 7455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.051775 7455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:26.051805 7455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.057636 7455 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:39601
I20260927 06:59:26.057678 7582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:39601 every 8 connection(s)
I20260927 06:59:26.057930 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:26.062489 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7455
I20260927 06:59:26.062593 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:26.062744 3300 external_mini_cluster.cc:1428] Setting key bfb773cbafbb3da93ba24263d1442665
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:26.067070 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.058595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:40871 (local address 127.3.57.62:46865)
0927 06:59:26.058668 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.058670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.058875 (+ 205us) server_negotiation.cc:407] Connection header received
0927 06:59:26.059315 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.059316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.059329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.059376 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.060278 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.060786 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.061700 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.061897 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.061950 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.064253 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.064255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.064255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.064263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.065714 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.066093 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.066094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.066095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.066155 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.066409 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.066409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.066410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.066532 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.066603 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.066870 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.066974 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 06:59:26.067215 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.058571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:46865 (local address 127.3.57.1:40871)
0927 06:59:26.058759 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.058793 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.059238 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.059433 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.059436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.059708 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:26.060199 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.060204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.060862 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.060862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.061632 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.061633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.061965 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.061966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.062055 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.062564 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:26.062573 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:26.064152 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:26.065782 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.065788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.065789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.065993 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.066230 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.066231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.066231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.066337 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.066605 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:26.066612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:26.066743 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.066846 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.066997 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":116,"threads_started":1}
I20260927 06:59:26.067752 7583 heartbeater.cc:347] Connected to a master server at 127.3.57.62:46865
I20260927 06:59:26.067839 7583 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:26.067992 7583 heartbeater.cc:511] Master 127.3.57.62:46865 requested a full tablet report, sending...
I20260927 06:59:26.068452 7393 ts_manager.cc:194] Registered new tserver with Master: aa23a7e6e219457ab68bdcd2a4ac2eb7 (127.3.57.1:39601)
I20260927 06:59:26.069015 7393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:40871
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:26.086072 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:26.165697 7588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.165877 7588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.165900 7588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.166965 7588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.167001 7588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.167025 7588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:26.168471 7588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.168638 7588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.168794 7588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:26.173404 7594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.173502 7588 server_base.cc:1034] running on GCE node
W20260927 06:59:26.173404 7593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.173485 7596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.173833 7588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.174079 7588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.175232 7588 hybrid_clock.cc:648] HybridClock initialized: now 1790492366175225 us; error 32 us; skew 500 ppm
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:26.177765 7588 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:26.178109 7588 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.178157 7588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.178293 7588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.179301 7588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "b483d4329be84998bba83bfdb4add7be"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "4286df79b943b4977598997ceed249b9"
server_key_iv: "2f0f8b648425c464627078c134bda226"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.179668 7588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "b483d4329be84998bba83bfdb4add7be"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "4286df79b943b4977598997ceed249b9"
server_key_iv: "2f0f8b648425c464627078c134bda226"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.180862 7588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:26.181519 7602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.181670 7588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.181757 7588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "b483d4329be84998bba83bfdb4add7be"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "4286df79b943b4977598997ceed249b9"
server_key_iv: "2f0f8b648425c464627078c134bda226"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.181833 7588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.200675 7588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.202350 7588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.202504 7588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.202703 7588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:26.203003 7588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:26.203053 7588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.203094 7588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:26.203119 7588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.208760 7588 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:36639
I20260927 06:59:26.208808 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:36639 every 8 connection(s)
I20260927 06:59:26.209046 7588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:26.212365 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7588
I20260927 06:59:26.212450 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:26.212652 3300 external_mini_cluster.cc:1428] Setting key 68acf55393699ebd5fb2b356c4f86393
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:26.217903 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.209648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:36867 (local address 127.3.57.62:46865)
0927 06:59:26.209688 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.209690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.209880 (+ 190us) server_negotiation.cc:407] Connection header received
0927 06:59:26.210316 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.210317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.210327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.210376 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.211214 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.211751 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.212803 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.213075 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.213137 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.215279 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.215281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.215282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.215288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.216610 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.216989 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.216990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.216991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.217049 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.217283 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.217284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.217284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.217389 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.217469 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.217715 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.217799 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 06:59:26.218072 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.209635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:46865 (local address 127.3.57.2:36867)
0927 06:59:26.209768 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.209799 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.210204 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.210423 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.210426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.210657 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:26.211114 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.211119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.211823 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.211823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.212689 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.212692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.213153 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.213154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.213287 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.213767 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:26.213774 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:26.215186 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:26.216696 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.216702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.216704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.216915 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.217120 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.217120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.217120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.217221 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.217461 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:26.217467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:26.217594 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.217694 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.217834 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":85,"threads_started":1}
I20260927 06:59:26.218518 7716 heartbeater.cc:347] Connected to a master server at 127.3.57.62:46865
I20260927 06:59:26.218598 7716 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:26.218729 7716 heartbeater.cc:511] Master 127.3.57.62:46865 requested a full tablet report, sending...
I20260927 06:59:26.219094 7393 ts_manager.cc:194] Registered new tserver with Master: b483d4329be84998bba83bfdb4add7be (127.3.57.2:36639)
I20260927 06:59:26.219525 7393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:36867
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:26.236395 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:26.312471 7721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.312656 7721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.312695 7721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.313714 7721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.313746 7721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.313771 7721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:26.315047 7721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:46865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.315228 7721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.315385 7721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:26.319942 7726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.319926 7729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.320070 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.320073 7721 server_base.cc:1034] running on GCE node
I20260927 06:59:26.320456 7721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.320695 7721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.321851 7721 hybrid_clock.cc:648] HybridClock initialized: now 1790492366321842 us; error 35 us; skew 500 ppm
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:26.324229 7721 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:26.324581 7721 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.324647 7721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.324788 7721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.325752 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "381e72c2e60c486c9f300e4befceb22b"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "3bd8f7b2e64669df20e15c37148933db"
server_key_iv: "e1f0f5cdca08da2997caafecc5f3e1eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.326099 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "381e72c2e60c486c9f300e4befceb22b"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "3bd8f7b2e64669df20e15c37148933db"
server_key_iv: "e1f0f5cdca08da2997caafecc5f3e1eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.327350 7721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:26.328102 7735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.328286 7721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.328377 7721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "381e72c2e60c486c9f300e4befceb22b"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "3bd8f7b2e64669df20e15c37148933db"
server_key_iv: "e1f0f5cdca08da2997caafecc5f3e1eb"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.328455 7721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.347019 7721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.349233 7721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.349381 7721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.349577 7721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:26.349865 7721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:26.349915 7721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.349953 7721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:26.349989 7721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.355808 7721 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:45995
I20260927 06:59:26.355841 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:45995 every 8 connection(s)
I20260927 06:59:26.356098 7721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:26.360564 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7721
I20260927 06:59:26.360682 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:26.360863 3300 external_mini_cluster.cc:1428] Setting key 11f2dd98cc6c43f50acb761d3ea319f1
Sep 27 06:59:26 dist-test-slave-mk3h krb5kdc[7349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492366, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:26.364537 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.356823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:37941 (local address 127.3.57.62:46865)
0927 06:59:26.356867 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.356869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.357044 (+ 175us) server_negotiation.cc:407] Connection header received
0927 06:59:26.357486 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.357486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.357495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.357550 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.358342 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.358829 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.359744 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.359926 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.359980 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.362081 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.362083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.362083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.362089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.363471 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.363841 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.363842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.363843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.363891 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.364086 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.364087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.364087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.364181 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.364229 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.364418 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.364476 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:26.364748 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.356781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:46865 (local address 127.3.57.3:37941)
0927 06:59:26.356928 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.356967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.357398 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.357595 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.357597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.357824 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:26.358265 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.358269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.358896 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.358896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.359682 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.359684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.359999 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.359999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.360085 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.360479 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:26.360489 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:26.362008 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:26.363535 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.363539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.363540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.363780 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.363946 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:26.363947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:26.363947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:26.364035 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:26.364230 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:26.364233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:26.364331 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.364406 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.364518 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 06:59:26.365206 7849 heartbeater.cc:347] Connected to a master server at 127.3.57.62:46865
I20260927 06:59:26.365302 7849 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:26.365439 7849 heartbeater.cc:511] Master 127.3.57.62:46865 requested a full tablet report, sending...
I20260927 06:59:26.365793 7393 ts_manager.cc:194] Registered new tserver with Master: 381e72c2e60c486c9f300e4befceb22b (127.3.57.3:45995)
I20260927 06:59:26.366140 7393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:37941
I20260927 06:59:26.372324 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:26.378958 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.373586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.3.57.62:46865)
0927 06:59:26.373632 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.373633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.373754 (+ 121us) server_negotiation.cc:407] Connection header received
0927 06:59:26.373798 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.373798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.373805 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.373849 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.374350 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.374783 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.375455 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.375699 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.375773 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.376781 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.376782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.376783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:26.376790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.377842 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.378167 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.378168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.378168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.378219 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:26.378413 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:26.378414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:26.378414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:26.378545 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.378641 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.378777 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.378852 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 06:59:26.380529 7393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:26.381978 7393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:26.388008 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.383951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39601 (local address 127.0.0.1:56038)
0927 06:59:26.384424 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.384436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.384493 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.384791 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.384792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.384794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:26.384858 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.384859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.386087 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.386088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.387020 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.387022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.387663 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.387753 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.387766 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.387855 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.387906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":46,"threads_started":1}
I20260927 06:59:26.388110 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.384027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.3.57.2:36639)
0927 06:59:26.384336 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.384341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.384351 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:26.384392 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.384395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.384411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.384512 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:26.384671 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.385820 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387021 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.387772 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387860 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.387906 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.387999 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.388033 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":67,"threads_started":1}
I20260927 06:59:26.388193 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.384078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.3.57.1:39601)
0927 06:59:26.384241 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.384245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.384498 (+ 253us) server_negotiation.cc:407] Connection header received
0927 06:59:26.384653 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.384655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.384671 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.384751 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:26.384938 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.385994 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387085 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.387594 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.387718 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.388064 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.388120 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260927 06:59:26.388367 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.383910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36639 (local address 127.0.0.1:52886)
0927 06:59:26.384092 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.384128 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.384193 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.384523 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.384525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.384528 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:26.384586 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.384588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.385926 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.385927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.386946 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.386949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.387873 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.387926 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.387930 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.387987 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.388018 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260927 06:59:26.389011 7650 tablet_service.cc:1467] Processing CreateTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 (DEFAULT_TABLE table=test-table [id=9d3f29438a0b48c5a29388b5c54e3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:26.389308 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.384130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:45995 (local address 127.0.0.1:45550)
0927 06:59:26.384860 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.384869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.384914 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.385141 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.385141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.385143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:26.385189 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.385191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.387112 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.387113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.388171 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.388175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.389078 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.389079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.389140 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.389144 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.389193 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.389231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":68,"threads_started":1}
I20260927 06:59:26.389258 7517 tablet_service.cc:1467] Processing CreateTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 (DEFAULT_TABLE table=test-table [id=9d3f29438a0b48c5a29388b5c54e3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:26.389362 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:26.389537 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:26.389540 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.384294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.3.57.3:45995)
0927 06:59:26.384406 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.384410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.384975 (+ 565us) server_negotiation.cc:407] Connection header received
0927 06:59:26.385014 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.385016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.385026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.385143 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:26.385269 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.385981 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.388268 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.388978 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.389074 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.389123 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.389432 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.389468 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260927 06:59:26.392697 7864 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Bootstrap starting.
I20260927 06:59:26.393404 7864 tablet_bootstrap.cc:654] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:26.393321 7783 tablet_service.cc:1467] Processing CreateTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 (DEFAULT_TABLE table=test-table [id=9d3f29438a0b48c5a29388b5c54e3335]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:26.393641 7783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:26.393996 7864 log.cc:824] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:26.394337 7863 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: Bootstrap starting.
I20260927 06:59:26.394707 7864 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: No bootstrap required, opened a new log
I20260927 06:59:26.394785 7864 ts_tablet_manager.cc:1397] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:26.394955 7863 tablet_bootstrap.cc:654] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:26.395529 7863 log.cc:824] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:26.396374 7863 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: No bootstrap required, opened a new log
I20260927 06:59:26.396449 7863 ts_tablet_manager.cc:1397] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 06:59:26.396540 7864 raft_consensus.cc:348] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.396703 7864 raft_consensus.cc:374] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:26.396764 7864 raft_consensus.cc:729] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b483d4329be84998bba83bfdb4add7be, State: Initialized, Role: FOLLOWER
I20260927 06:59:26.396916 7864 consensus_queue.cc:260] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.397110 7867 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Bootstrap starting.
I20260927 06:59:26.397208 7864 ts_tablet_manager.cc:1428] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:26.397415 7716 heartbeater.cc:503] Master 127.3.57.62:46865 was elected leader, sending a full tablet report...
I20260927 06:59:26.397756 7867 tablet_bootstrap.cc:654] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:26.398401 7867 log.cc:824] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:26.398533 7863 raft_consensus.cc:348] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.398684 7863 raft_consensus.cc:374] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:26.398846 7863 raft_consensus.cc:729] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa23a7e6e219457ab68bdcd2a4ac2eb7, State: Initialized, Role: FOLLOWER
I20260927 06:59:26.399181 7867 tablet_bootstrap.cc:492] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: No bootstrap required, opened a new log
I20260927 06:59:26.399240 7867 ts_tablet_manager.cc:1397] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:26.399020 7863 consensus_queue.cc:260] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.399484 7863 ts_tablet_manager.cc:1428] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 06:59:26.400638 7867 raft_consensus.cc:348] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.400753 7867 raft_consensus.cc:374] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:26.400775 7867 raft_consensus.cc:729] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381e72c2e60c486c9f300e4befceb22b, State: Initialized, Role: FOLLOWER
I20260927 06:59:26.400844 7867 consensus_queue.cc:260] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.401060 7867 ts_tablet_manager.cc:1428] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 06:59:26.401074 7849 heartbeater.cc:503] Master 127.3.57.62:46865 was elected leader, sending a full tablet report...
I20260927 06:59:26.401196 7583 heartbeater.cc:503] Master 127.3.57.62:46865 was elected leader, sending a full tablet report...
W20260927 06:59:26.459702 7717 tablet.cc:2369] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:26.501389 7868 raft_consensus.cc:482] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:26.501513 7868 raft_consensus.cc:504] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.502318 7868 leader_election.cc:288] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa23a7e6e219457ab68bdcd2a4ac2eb7 (127.3.57.1:39601), 381e72c2e60c486c9f300e4befceb22b (127.3.57.3:45995)
I20260927 06:59:26.504906 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.502215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39601 (local address 127.3.57.2:44921)
0927 06:59:26.502282 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.502295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.502341 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.502513 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.502514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.502516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:26.502599 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.502601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.503278 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.503279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.504132 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.504134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.504656 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.504737 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.504746 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.504788 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.504818 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":33}
I20260927 06:59:26.504964 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.502328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44921 (local address 127.3.57.1:39601)
0927 06:59:26.502374 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.502376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.502388 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:26.502417 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.502418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.502430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.502491 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:26.502672 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.503199 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504227 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.504580 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504633 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.504641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.504812 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.504861 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:26.504982 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.502285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:40275 (local address 127.3.57.3:45995)
0927 06:59:26.502417 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.502419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.502507 (+ 88us) server_negotiation.cc:407] Connection header received
0927 06:59:26.502554 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.502555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.502568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.502629 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:26.502804 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.503259 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504226 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.504629 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504696 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.504706 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.504871 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.504909 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 06:59:26.505034 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.502221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:45995 (local address 127.3.57.2:40275)
0927 06:59:26.502432 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.502442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.502498 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.502650 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.502651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.502653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:26.502735 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.502736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.503343 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.503343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.504134 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:26.504136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.504715 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.504716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:26.504791 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.504795 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.504853 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.504887 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260927 06:59:26.505419 7537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18b00c8a78c4b4f8e75bd97e2d4d8f9" candidate_uuid: "b483d4329be84998bba83bfdb4add7be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" is_pre_election: true
I20260927 06:59:26.505529 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18b00c8a78c4b4f8e75bd97e2d4d8f9" candidate_uuid: "b483d4329be84998bba83bfdb4add7be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381e72c2e60c486c9f300e4befceb22b" is_pre_election: true
I20260927 06:59:26.505590 7537 raft_consensus.cc:2454] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b483d4329be84998bba83bfdb4add7be in term 0.
I20260927 06:59:26.505695 7803 raft_consensus.cc:2454] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b483d4329be84998bba83bfdb4add7be in term 0.
I20260927 06:59:26.505844 7606 leader_election.cc:302] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa23a7e6e219457ab68bdcd2a4ac2eb7, b483d4329be84998bba83bfdb4add7be; no voters:
I20260927 06:59:26.505975 7868 raft_consensus.cc:2781] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:26.506029 7868 raft_consensus.cc:482] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:26.506062 7868 raft_consensus.cc:3037] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:26.506740 7868 raft_consensus.cc:504] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.506875 7868 leader_election.cc:288] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [CANDIDATE]: Term 1 election: Requested vote from peers aa23a7e6e219457ab68bdcd2a4ac2eb7 (127.3.57.1:39601), 381e72c2e60c486c9f300e4befceb22b (127.3.57.3:45995)
I20260927 06:59:26.507037 7537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18b00c8a78c4b4f8e75bd97e2d4d8f9" candidate_uuid: "b483d4329be84998bba83bfdb4add7be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7"
I20260927 06:59:26.507036 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18b00c8a78c4b4f8e75bd97e2d4d8f9" candidate_uuid: "b483d4329be84998bba83bfdb4add7be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381e72c2e60c486c9f300e4befceb22b"
I20260927 06:59:26.507110 7537 raft_consensus.cc:3037] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:26.507110 7803 raft_consensus.cc:3037] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:26.508008 7803 raft_consensus.cc:2454] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b483d4329be84998bba83bfdb4add7be in term 1.
I20260927 06:59:26.508014 7537 raft_consensus.cc:2454] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b483d4329be84998bba83bfdb4add7be in term 1.
I20260927 06:59:26.508198 7604 leader_election.cc:302] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381e72c2e60c486c9f300e4befceb22b, b483d4329be84998bba83bfdb4add7be; no voters:
I20260927 06:59:26.508320 7868 raft_consensus.cc:2781] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:26.508493 7868 raft_consensus.cc:686] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 1 LEADER]: Becoming Leader. State: Replica: b483d4329be84998bba83bfdb4add7be, State: Running, Role: LEADER
I20260927 06:59:26.508589 7868 consensus_queue.cc:237] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } }
I20260927 06:59:26.509339 7392 catalog_manager.cc:5495] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be reported cstate change: term changed from 0 to 1, leader changed from <none> to b483d4329be84998bba83bfdb4add7be (127.3.57.2). New cstate: current_term: 1 leader_uuid: "b483d4329be84998bba83bfdb4add7be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b483d4329be84998bba83bfdb4add7be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 36639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 } health_report { overall_health: UNKNOWN } } }
I20260927 06:59:26.534619 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.532552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.3.57.2:36639)
0927 06:59:26.532593 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.532595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.532603 (+ 8us) server_negotiation.cc:407] Connection header received
0927 06:59:26.532679 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.532679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.532694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.532752 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:26.532909 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.533334 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.534017 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:26.534182 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:26.534228 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:26.534359 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:26.534451 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:26.534509 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.534539 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.534554 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:26.536305 7537 raft_consensus.cc:1261] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 1 FOLLOWER]: Refusing update from remote peer b483d4329be84998bba83bfdb4add7be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:26.536309 7803 raft_consensus.cc:1261] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 1 FOLLOWER]: Refusing update from remote peer b483d4329be84998bba83bfdb4add7be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:26.536646 7868 consensus_queue.cc:1035] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "381e72c2e60c486c9f300e4befceb22b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:26.536762 7873 consensus_queue.cc:1035] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa23a7e6e219457ab68bdcd2a4ac2eb7" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:26.539614 7877 mvcc.cc:205] Tried to move back new op lower bound from 7333856733330132992 to 7333856733219700736. Current Snapshot: MvccSnapshot[applied={T|T < 7333856733330132992}]
I20260927 06:59:26.541054 7393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:26.541131 7393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:26.541862 7393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4bf5018f091441194d37024ef1edeef: Sending DeleteTablet for 3 replicas of tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9
I20260927 06:59:26.542168 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:26 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:52886
I20260927 06:59:26.542201 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:26 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:56038
I20260927 06:59:26.542239 7783 tablet_service.cc:1514] Processing DeleteTablet for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:26 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:45550
I20260927 06:59:26.542490 7887 tablet_replica.cc:330] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7: stopping tablet replica
I20260927 06:59:26.542526 7888 tablet_replica.cc:330] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: stopping tablet replica
I20260927 06:59:26.542621 7887 raft_consensus.cc:2229] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P aa23a7e6e219457ab68bdcd2a4ac2eb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:26.542642 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7455
I20260927 06:59:26.543531 7888 raft_consensus.cc:2229] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:26.543679 7888 raft_consensus.cc:2258] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:26.544064 7888 ts_tablet_manager.cc:1905] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:26.545444 7888 ts_tablet_manager.cc:1918] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:26.545526 7888 log.cc:1196] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/b18b00c8a78c4b4f8e75bd97e2d4d8f9
I20260927 06:59:26.545763 7888 ts_tablet_manager.cc:1939] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P 381e72c2e60c486c9f300e4befceb22b: Deleting consensus metadata
I20260927 06:59:26.546135 7378 catalog_manager.cc:4841] TS 381e72c2e60c486c9f300e4befceb22b (127.3.57.3:45995): tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9 (table test-table [id=9d3f29438a0b48c5a29388b5c54e3335]) successfully deleted
I20260927 06:59:26.547662 7886 tablet_replica.cc:330] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: stopping tablet replica
I20260927 06:59:26.547765 7886 raft_consensus.cc:2229] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:26.547855 7886 raft_consensus.cc:2258] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:26.548225 7886 ts_tablet_manager.cc:1905] T b18b00c8a78c4b4f8e75bd97e2d4d8f9 P b483d4329be84998bba83bfdb4add7be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 06:59:26.548907 7380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:39601 (error 108)
I20260927 06:59:26.549352 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.549235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39601 (local address 127.0.0.1:56040)
0927 06:59:26.549279 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.549315 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:39601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 06:59:26.549403 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7588
W20260927 06:59:26.549556 7380 catalog_manager.cc:4584] TS aa23a7e6e219457ab68bdcd2a4ac2eb7 (127.3.57.1:39601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b18b00c8a78c4b4f8e75bd97e2d4d8f9: Network error: Client connection negotiation failed: client connection to 127.3.57.1:39601: connect: Connection refused (error 111)
I20260927 06:59:26.554049 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.553950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:36639 (local address 127.0.0.1:52892)
0927 06:59:26.553995 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.554016 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:36639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 06:59:26.554472 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7721
I20260927 06:59:26.559036 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7362
2026-09-27T06:59:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2476 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T06:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:26Z Disabled control of system clock
I20260927 06:59:26.585505 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:43027
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:43027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:59:26.664592 7893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.664790 7893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.664811 7893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.665892 7893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:26.665932 7893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.665946 7893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.665957 7893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:26.665969 7893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:26.667234 7893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:43027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:43027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.667428 7893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.667713 7893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:26.672317 7898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.672363 7893 server_base.cc:1034] running on GCE node
W20260927 06:59:26.672472 7899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.672372 7901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.672760 7893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.673032 7893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.674177 7893 hybrid_clock.cc:648] HybridClock initialized: now 1790492366674155 us; error 37 us; skew 500 ppm
I20260927 06:59:26.675354 7893 webserver.cc:460] Webserver started at http://127.3.57.62:41835/ using document root <none> and password file <none>
I20260927 06:59:26.675606 7893 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.675660 7893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.675747 7893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.676702 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "f36378f4bf254a329060965d1ef7619c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "1a61edf0916b67c205afce56828ed126"
server_key_iv: "2cedcaea5b4e5b2d0f54106350250b39"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.676998 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "f36378f4bf254a329060965d1ef7619c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "1a61edf0916b67c205afce56828ed126"
server_key_iv: "2cedcaea5b4e5b2d0f54106350250b39"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.678144 7893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:26.678772 7907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.678934 7893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:26.679013 7893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "f36378f4bf254a329060965d1ef7619c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "1a61edf0916b67c205afce56828ed126"
server_key_iv: "2cedcaea5b4e5b2d0f54106350250b39"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.679095 7893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.682163 7893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.682370 7893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.682462 7893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.686198 7893 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:43027
I20260927 06:59:26.686297 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:43027 every 8 connection(s)
I20260927 06:59:26.686623 7893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:26.687603 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:26.689301 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7893
I20260927 06:59:26.689399 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:26.689587 3300 external_mini_cluster.cc:1428] Setting key 304bc7dabb414de82f85e47ca8a4fb0c
I20260927 06:59:26.691022 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: Bootstrap starting.
I20260927 06:59:26.691512 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:26.691912 7960 log.cc:824] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:26.692658 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: No bootstrap required, opened a new log
I20260927 06:59:26.693660 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.690087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.3.57.62:43027)
0927 06:59:26.691736 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.691765 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.691799 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:59:26.692314 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.692323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.692374 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.692766 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.693233 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.693236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.693237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:26.693241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.693283 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:26.693298 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.693376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.693412 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.693467 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":57,"threads_started":1}
I20260927 06:59:26.694550 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } }
I20260927 06:59:26.694672 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:26.694693 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36378f4bf254a329060965d1ef7619c, State: Initialized, Role: FOLLOWER
I20260927 06:59:26.694797 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } }
I20260927 06:59:26.694880 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:26.694902 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:26.694931 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:26.695725 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } }
I20260927 06:59:26.695809 7960 leader_election.cc:302] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36378f4bf254a329060965d1ef7619c; no voters:
I20260927 06:59:26.695923 7960 leader_election.cc:288] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:26.695992 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:26.696125 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [term 1 LEADER]: Becoming Leader. State: Replica: f36378f4bf254a329060965d1ef7619c, State: Running, Role: LEADER
I20260927 06:59:26.696166 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:26.696246 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } }
I20260927 06:59:26.696606 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36378f4bf254a329060965d1ef7619c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } } }
I20260927 06:59:26.696610 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36378f4bf254a329060965d1ef7619c. Latest consensus state: current_term: 1 leader_uuid: "f36378f4bf254a329060965d1ef7619c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36378f4bf254a329060965d1ef7619c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43027 } } }
I20260927 06:59:26.696751 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:26.696761 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:26.697263 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:26.697726 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:26.699128 7974 catalog_manager.cc:1347] Generated new cluster ID: 803d6b43e7ed42c6965ffa6c334f4232
I20260927 06:59:26.699184 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:26.704721 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:26.704866 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:26.711484 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: Generated new TSK 0
I20260927 06:59:26.711685 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:26.714228 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--builtin_ntp_servers=127.3.57.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:59:26.806450 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.806627 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.806648 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.807847 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.807884 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.807909 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:26.809226 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.809435 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.809718 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:26.814358 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.814339 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.814517 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.814450 7984 server_base.cc:1034] running on GCE node
I20260927 06:59:26.814682 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.814926 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.816062 7984 hybrid_clock.cc:648] HybridClock initialized: now 1790492366816048 us; error 31 us; skew 500 ppm
I20260927 06:59:26.817258 7984 webserver.cc:460] Webserver started at http://127.3.57.1:45401/ using document root <none> and password file <none>
I20260927 06:59:26.817502 7984 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.817570 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.817701 7984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.818683 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "69cd123fd4e24f3493c475b9a6457fb8"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "7f6121692bd1dbac1fc0c233a5bd8af9"
server_key_iv: "8abc2b1e88d51d8918019f525795f5e0"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.819053 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "69cd123fd4e24f3493c475b9a6457fb8"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "7f6121692bd1dbac1fc0c233a5bd8af9"
server_key_iv: "8abc2b1e88d51d8918019f525795f5e0"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.820449 7984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:26.821246 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.821408 7984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:26.821499 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "69cd123fd4e24f3493c475b9a6457fb8"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "7f6121692bd1dbac1fc0c233a5bd8af9"
server_key_iv: "8abc2b1e88d51d8918019f525795f5e0"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.821570 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.824784 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.825026 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.825160 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.825357 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:26.825634 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:26.825682 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.825721 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:26.825754 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.832329 7984 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:35465
I20260927 06:59:26.832370 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:35465 every 8 connection(s)
I20260927 06:59:26.832720 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:26.835798 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.833447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:50779 (local address 127.3.57.62:43027)
0927 06:59:26.833487 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.833489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.833721 (+ 232us) server_negotiation.cc:407] Connection header received
0927 06:59:26.834223 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.834224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.834232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.834292 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.835513 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.835515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.835516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:26.835521 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.835537 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:26.835542 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.835643 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.835716 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.835731 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 06:59:26.836005 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.833435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:43027 (local address 127.3.57.1:50779)
0927 06:59:26.833609 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.833641 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.834111 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.834358 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.834362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.834372 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:26.835343 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:26.835343 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:26.835371 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:26.835397 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:26.835646 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:26.835652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:26.835655 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.835728 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.835787 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260927 06:59:26.836454 8112 heartbeater.cc:347] Connected to a master server at 127.3.57.62:43027
I20260927 06:59:26.836537 8112 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:26.836694 8112 heartbeater.cc:511] Master 127.3.57.62:43027 requested a full tablet report, sending...
I20260927 06:59:26.837179 7924 ts_manager.cc:194] Registered new tserver with Master: 69cd123fd4e24f3493c475b9a6457fb8 (127.3.57.1:35465)
I20260927 06:59:26.840894 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 7984
I20260927 06:59:26.841004 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:26.841169 3300 external_mini_cluster.cc:1428] Setting key 554b0b4301fbf18635eae8198f97a0d3
I20260927 06:59:26.841955 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--builtin_ntp_servers=127.3.57.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:59:26.920619 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:26.920859 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:26.920895 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:26.922060 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:26.922125 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:26.922169 8115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:26.923497 8115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:26.923775 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:26.924083 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:59:26.929011 8115 server_base.cc:1034] running on GCE node
W20260927 06:59:26.929033 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.929075 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:26.929139 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:26.929430 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:26.929679 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:26.930826 8115 hybrid_clock.cc:648] HybridClock initialized: now 1790492366930813 us; error 26 us; skew 500 ppm
I20260927 06:59:26.932026 8115 webserver.cc:460] Webserver started at http://127.3.57.2:39319/ using document root <none> and password file <none>
I20260927 06:59:26.932252 8115 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:26.932315 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:26.932443 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:26.933432 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "76042b862ffe447cbcd4d5be33fb187c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "63cf820da39e01ecec376438f34aab9d"
server_key_iv: "dfb68e4f5c0b247ba45fca7527f75e38"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.933805 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "76042b862ffe447cbcd4d5be33fb187c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "63cf820da39e01ecec376438f34aab9d"
server_key_iv: "dfb68e4f5c0b247ba45fca7527f75e38"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.935091 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 06:59:26.935853 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.936022 8115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:26.936115 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "76042b862ffe447cbcd4d5be33fb187c"
format_stamp: "Formatted at 2026-09-27 06:59:26 on dist-test-slave-mk3h"
server_key: "63cf820da39e01ecec376438f34aab9d"
server_key_iv: "dfb68e4f5c0b247ba45fca7527f75e38"
server_key_version: "encryptionkey@0"
I20260927 06:59:26.936187 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:26.939347 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:26.939640 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:26.939774 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:26.940006 8115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:26.940301 8115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:26.940351 8115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.940393 8115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:26.940420 8115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:26.945917 8115 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:34373
I20260927 06:59:26.945952 8242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:34373 every 8 connection(s)
I20260927 06:59:26.946269 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:26.948984 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.946968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:52687 (local address 127.3.57.62:43027)
0927 06:59:26.947027 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:26.947029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:26.947231 (+ 202us) server_negotiation.cc:407] Connection header received
0927 06:59:26.947708 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:26.947709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:26.947713 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:26.947776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:26.948791 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:26.948792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:26.948792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:26.948796 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:26.948810 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:26.948815 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:26.948867 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:26.948913 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 06:59:26.948925 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 06:59:26.949180 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:26.946948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:43027 (local address 127.3.57.2:52687)
0927 06:59:26.947113 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:26.947150 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:26.947620 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:26.947826 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:26.947829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:26.947837 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:26.948660 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:26.948660 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:26.948685 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:26.948711 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:26.948866 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:26.948869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:26.948871 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:26.948927 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 06:59:26.948970 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260927 06:59:26.949630 8243 heartbeater.cc:347] Connected to a master server at 127.3.57.62:43027
I20260927 06:59:26.949705 8243 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:26.949810 8243 heartbeater.cc:511] Master 127.3.57.62:43027 requested a full tablet report, sending...
I20260927 06:59:26.950102 7924 ts_manager.cc:194] Registered new tserver with Master: 76042b862ffe447cbcd4d5be33fb187c (127.3.57.2:34373)
I20260927 06:59:26.956139 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8115
I20260927 06:59:26.956259 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:26.956458 3300 external_mini_cluster.cc:1428] Setting key 49e5a82789b42bc6c61d4e12d96081b7
I20260927 06:59:26.957319 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--builtin_ntp_servers=127.3.57.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 06:59:27.036233 8246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:27.036450 8246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:27.036473 8246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:27.037662 8246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:27.037698 8246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:27.037745 8246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:27.039125 8246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:43027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:27.039350 8246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:27.039691 8246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:27.044308 8251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.044318 8254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.044339 8252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.044525 8246 server_base.cc:1034] running on GCE node
I20260927 06:59:27.044725 8246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:27.044960 8246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:27.046149 8246 hybrid_clock.cc:648] HybridClock initialized: now 1790492367046133 us; error 28 us; skew 500 ppm
I20260927 06:59:27.047381 8246 webserver.cc:460] Webserver started at http://127.3.57.3:44833/ using document root <none> and password file <none>
I20260927 06:59:27.047659 8246 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:27.047724 8246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:27.047852 8246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:27.048810 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "0ec3ef075ade48bbad31c027028e794b"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "cb9af1ae3352d4c272ad15d344780198"
server_key_iv: "4b06e90aa9bca3961df1fc6a71ef512a"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.049137 8246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "0ec3ef075ade48bbad31c027028e794b"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "cb9af1ae3352d4c272ad15d344780198"
server_key_iv: "4b06e90aa9bca3961df1fc6a71ef512a"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.050369 8246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:27.051075 8260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.051275 8246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:27.051364 8246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "0ec3ef075ade48bbad31c027028e794b"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "cb9af1ae3352d4c272ad15d344780198"
server_key_iv: "4b06e90aa9bca3961df1fc6a71ef512a"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.051442 8246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:27.054533 8246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:27.054782 8246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:27.054908 8246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:27.055099 8246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:27.055399 8246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:27.055449 8246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.055488 8246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:27.055516 8246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.060760 8246 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:46543
I20260927 06:59:27.060793 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:46543 every 8 connection(s)
I20260927 06:59:27.061074 8246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:27.061465 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8246
I20260927 06:59:27.061560 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:27.061755 3300 external_mini_cluster.cc:1428] Setting key e1b0db841978fee858873ff96e522bb2
I20260927 06:59:27.064090 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.061936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:50893 (local address 127.3.57.62:43027)
0927 06:59:27.061983 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.061985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.062084 (+ 99us) server_negotiation.cc:407] Connection header received
0927 06:59:27.062642 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.062642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.062648 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.062691 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.063849 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.063851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.063851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.063855 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.063871 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.063875 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.063928 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.064009 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.064023 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 06:59:27.064273 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.061791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:43027 (local address 127.3.57.3:50893)
0927 06:59:27.061969 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.062008 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.062557 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.062749 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.062752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.062761 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.063711 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.063711 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.063735 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.063761 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.063928 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.063933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.063935 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.064018 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.064072 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260927 06:59:27.064704 8374 heartbeater.cc:347] Connected to a master server at 127.3.57.62:43027
I20260927 06:59:27.064798 8374 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:27.064895 8374 heartbeater.cc:511] Master 127.3.57.62:43027 requested a full tablet report, sending...
I20260927 06:59:27.065186 7924 ts_manager.cc:194] Registered new tserver with Master: 0ec3ef075ade48bbad31c027028e794b (127.3.57.3:46543)
I20260927 06:59:27.073426 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 06:59:27.075533 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.074642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.3.57.62:43027)
0927 06:59:27.074694 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.074695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.074771 (+ 76us) server_negotiation.cc:407] Connection header received
0927 06:59:27.074842 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.074842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.074846 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.074884 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.075389 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.075390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.075390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.075393 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.075405 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.075408 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.075446 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.075478 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.075489 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 06:59:27.076872 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:27.078382 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:27.081920 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:46543 (local address 127.0.0.1:55858)
0927 06:59:27.080430 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.080470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.080526 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.080976 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.080981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.080987 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.081495 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.081495 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.081509 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.081531 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.081758 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.081764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.081767 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.081833 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.081847 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260927 06:59:27.082190 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.3.57.3:46543)
0927 06:59:27.080827 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.080831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.080841 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:27.080866 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.080868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.080875 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.080961 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.081622 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.081629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.081629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.081636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.081667 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.081687 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.081762 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.082078 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.082096 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":87,"threads_started":1}
I20260927 06:59:27.082297 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:35465 (local address 127.0.0.1:41112)
0927 06:59:27.080594 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.080604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.080659 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.081014 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.081014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.081016 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.081766 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.081766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.081773 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.081782 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.082175 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.082176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.082176 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.082227 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.082238 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":78,"threads_started":1}
I20260927 06:59:27.082341 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.3.57.1:35465)
0927 06:59:27.080583 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.080588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.080651 (+ 63us) server_negotiation.cc:407] Connection header received
0927 06:59:27.080728 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.080732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.080744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.080826 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.082026 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.082030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.082030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.082034 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.082105 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.082121 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.082182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.082231 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.082245 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260927 06:59:27.082083 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34373 (local address 127.0.0.1:51408)
0927 06:59:27.080673 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.080681 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.080716 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.081327 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.081328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.081330 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.081735 (+ 405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.081736 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.081742 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.081754 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.081940 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.081958 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.081959 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.082009 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.082021 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":62,"threads_started":1}
I20260927 06:59:27.083019 8308 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39f9a944941f4949fe3e446b1431f (DEFAULT_TABLE table=test-table [id=86163fb68f454d83a75f6e8b65472df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:27.083024 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39f9a944941f4949fe3e446b1431f (DEFAULT_TABLE table=test-table [id=86163fb68f454d83a75f6e8b65472df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:27.083359 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39f9a944941f4949fe3e446b1431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.083359 8308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39f9a944941f4949fe3e446b1431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.086542 8389 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: Bootstrap starting.
I20260927 06:59:27.086719 8388 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Bootstrap starting.
I20260927 06:59:27.086777 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.080677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.3.57.2:34373)
0927 06:59:27.080837 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.080841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.080851 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:27.080874 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.080876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.080884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.080964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.081820 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.081826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.081826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.081830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.081858 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.081884 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.081963 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.082205 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.082223 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260927 06:59:27.087294 8388 tablet_bootstrap.cc:654] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.087293 8389 tablet_bootstrap.cc:654] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.087298 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb39f9a944941f4949fe3e446b1431f (DEFAULT_TABLE table=test-table [id=86163fb68f454d83a75f6e8b65472df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:27.087652 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb39f9a944941f4949fe3e446b1431f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.087666 8389 log.cc:824] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.087666 8388 log.cc:824] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.088373 8388 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: No bootstrap required, opened a new log
I20260927 06:59:27.088408 8389 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: No bootstrap required, opened a new log
I20260927 06:59:27.088452 8388 ts_tablet_manager.cc:1397] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:27.088465 8389 ts_tablet_manager.cc:1397] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 06:59:27.090163 8388 raft_consensus.cc:348] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.090328 8388 raft_consensus.cc:374] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.090372 8388 raft_consensus.cc:729] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec3ef075ade48bbad31c027028e794b, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.090512 8388 consensus_queue.cc:260] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.090778 8388 ts_tablet_manager.cc:1428] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:27.090818 8374 heartbeater.cc:503] Master 127.3.57.62:43027 was elected leader, sending a full tablet report...
I20260927 06:59:27.091035 8393 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: Bootstrap starting.
I20260927 06:59:27.091013 8389 raft_consensus.cc:348] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.091169 8389 raft_consensus.cc:374] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.091204 8389 raft_consensus.cc:729] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69cd123fd4e24f3493c475b9a6457fb8, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.091318 8389 consensus_queue.cc:260] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.091598 8389 ts_tablet_manager.cc:1428] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 06:59:27.091714 8393 tablet_bootstrap.cc:654] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.091764 8112 heartbeater.cc:503] Master 127.3.57.62:43027 was elected leader, sending a full tablet report...
I20260927 06:59:27.092031 8393 log.cc:824] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.092800 8393 tablet_bootstrap.cc:492] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: No bootstrap required, opened a new log
I20260927 06:59:27.092851 8393 ts_tablet_manager.cc:1397] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:27.094770 8393 raft_consensus.cc:348] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.094884 8393 raft_consensus.cc:374] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.094904 8393 raft_consensus.cc:729] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76042b862ffe447cbcd4d5be33fb187c, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.095006 8393 consensus_queue.cc:260] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.095202 8393 ts_tablet_manager.cc:1428] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 06:59:27.095261 8243 heartbeater.cc:503] Master 127.3.57.62:43027 was elected leader, sending a full tablet report...
I20260927 06:59:27.150689 8394 raft_consensus.cc:482] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:27.150811 8394 raft_consensus.cc:504] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.151202 8394 leader_election.cc:288] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec3ef075ade48bbad31c027028e794b (127.3.57.3:46543), 76042b862ffe447cbcd4d5be33fb187c (127.3.57.2:34373)
I20260927 06:59:27.152680 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.151375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:46543 (local address 127.3.57.1:54317)
0927 06:59:27.151510 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.151521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.151601 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.151783 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.151784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.151787 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.152275 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.152276 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.152307 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.152323 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.152512 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.152513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.152514 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.152585 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.152599 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 06:59:27.152715 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.151447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:54317 (local address 127.3.57.3:46543)
0927 06:59:27.151532 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.151534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.151581 (+ 47us) server_negotiation.cc:407] Connection header received
0927 06:59:27.151686 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.151687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.151693 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.151776 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.152406 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.152407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.152407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.152411 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.152441 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.152446 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.152508 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.152594 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.152608 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 06:59:27.152818 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.151669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:33429 (local address 127.3.57.2:34373)
0927 06:59:27.151719 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.151720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.151812 (+ 92us) server_negotiation.cc:407] Connection header received
0927 06:59:27.151866 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.151867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.151872 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.151924 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.152573 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.152574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.152575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.152579 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.152597 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.152601 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.152656 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.152711 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.152725 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:27.153012 8328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39f9a944941f4949fe3e446b1431f" candidate_uuid: "69cd123fd4e24f3493c475b9a6457fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec3ef075ade48bbad31c027028e794b" is_pre_election: true
I20260927 06:59:27.153167 8328 raft_consensus.cc:2454] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69cd123fd4e24f3493c475b9a6457fb8 in term 0.
I20260927 06:59:27.153331 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.151515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34373 (local address 127.3.57.1:33429)
0927 06:59:27.151738 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.151747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.151812 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.151932 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.151933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.151935 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:27.152473 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 06:59:27.152473 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:27.152480 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 06:59:27.152494 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:27.152661 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:27.152661 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:27.152662 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 06:59:27.152710 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 06:59:27.152724 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":140,"threads_started":1}
I20260927 06:59:27.153502 8002 leader_election.cc:302] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec3ef075ade48bbad31c027028e794b, 69cd123fd4e24f3493c475b9a6457fb8; no voters:
I20260927 06:59:27.153644 8394 raft_consensus.cc:2781] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:27.153752 8394 raft_consensus.cc:482] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:27.153784 8394 raft_consensus.cc:3037] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.153872 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39f9a944941f4949fe3e446b1431f" candidate_uuid: "69cd123fd4e24f3493c475b9a6457fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76042b862ffe447cbcd4d5be33fb187c" is_pre_election: true
I20260927 06:59:27.154006 8197 raft_consensus.cc:2454] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69cd123fd4e24f3493c475b9a6457fb8 in term 0.
I20260927 06:59:27.154668 8394 raft_consensus.cc:504] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.154788 8394 leader_election.cc:288] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec3ef075ade48bbad31c027028e794b (127.3.57.3:46543), 76042b862ffe447cbcd4d5be33fb187c (127.3.57.2:34373)
I20260927 06:59:27.154947 8328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39f9a944941f4949fe3e446b1431f" candidate_uuid: "69cd123fd4e24f3493c475b9a6457fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec3ef075ade48bbad31c027028e794b"
I20260927 06:59:27.154968 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb39f9a944941f4949fe3e446b1431f" candidate_uuid: "69cd123fd4e24f3493c475b9a6457fb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76042b862ffe447cbcd4d5be33fb187c"
I20260927 06:59:27.155021 8328 raft_consensus.cc:3037] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.155049 8197 raft_consensus.cc:3037] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.155766 8197 raft_consensus.cc:2454] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69cd123fd4e24f3493c475b9a6457fb8 in term 1.
I20260927 06:59:27.155784 8328 raft_consensus.cc:2454] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69cd123fd4e24f3493c475b9a6457fb8 in term 1.
I20260927 06:59:27.155922 8002 leader_election.cc:302] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69cd123fd4e24f3493c475b9a6457fb8, 76042b862ffe447cbcd4d5be33fb187c; no voters:
I20260927 06:59:27.156018 8394 raft_consensus.cc:2781] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:27.156167 8394 raft_consensus.cc:686] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 69cd123fd4e24f3493c475b9a6457fb8, State: Running, Role: LEADER
I20260927 06:59:27.156275 8394 consensus_queue.cc:237] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } }
I20260927 06:59:27.157054 7924 catalog_manager.cc:5495] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69cd123fd4e24f3493c475b9a6457fb8 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "69cd123fd4e24f3493c475b9a6457fb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69cd123fd4e24f3493c475b9a6457fb8" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 } health_report { overall_health: UNKNOWN } } }
I20260927 06:59:27.160311 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.159514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.3.57.1:35465)
0927 06:59:27.159572 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.159574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.159587 (+ 13us) server_negotiation.cc:407] Connection header received
0927 06:59:27.159640 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.159641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.159649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.159693 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.160173 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.160174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.160174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.160177 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.160190 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.160193 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.160236 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.160264 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.160274 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 06:59:27.161734 8197 raft_consensus.cc:1261] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 1 FOLLOWER]: Refusing update from remote peer 69cd123fd4e24f3493c475b9a6457fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:27.161757 8328 raft_consensus.cc:1261] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 1 FOLLOWER]: Refusing update from remote peer 69cd123fd4e24f3493c475b9a6457fb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:27.162041 8394 consensus_queue.cc:1035] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76042b862ffe447cbcd4d5be33fb187c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:27.162178 8398 consensus_queue.cc:1035] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec3ef075ade48bbad31c027028e794b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:27.164350 8401 mvcc.cc:205] Tried to move back new op lower bound from 7333856735892242432 to 7333856735872819200. Current Snapshot: MvccSnapshot[applied={T|T < 7333856735892242432}]
I20260927 06:59:27.165059 8403 mvcc.cc:205] Tried to move back new op lower bound from 7333856735892242432 to 7333856735872819200. Current Snapshot: MvccSnapshot[applied={T|T < 7333856735892242432}]
I20260927 06:59:27.167026 7924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:27.167119 7924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:27.167799 7924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f36378f4bf254a329060965d1ef7619c: Sending DeleteTablet for 3 replicas of tablet 8bb39f9a944941f4949fe3e446b1431f
I20260927 06:59:27.168133 8308 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39f9a944941f4949fe3e446b1431f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:27 UTC) from {username='slave'} at 127.0.0.1:55858
I20260927 06:59:27.168211 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39f9a944941f4949fe3e446b1431f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:27 UTC) from {username='slave'} at 127.0.0.1:41112
I20260927 06:59:27.168392 8410 tablet_replica.cc:330] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: stopping tablet replica
I20260927 06:59:27.168380 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb39f9a944941f4949fe3e446b1431f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:27 UTC) from {username='slave'} at 127.0.0.1:51408
I20260927 06:59:27.168500 8410 raft_consensus.cc:2229] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:27.168599 8410 raft_consensus.cc:2258] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:27.168661 8411 tablet_replica.cc:330] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8: stopping tablet replica
I20260927 06:59:27.168747 8411 raft_consensus.cc:2229] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:27.168866 8411 raft_consensus.cc:2258] T 8bb39f9a944941f4949fe3e446b1431f P 69cd123fd4e24f3493c475b9a6457fb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:27.168886 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7984
I20260927 06:59:27.169016 8410 ts_tablet_manager.cc:1905] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:27.170646 8410 ts_tablet_manager.cc:1918] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:27.170696 8410 log.cc:1196] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/8bb39f9a944941f4949fe3e446b1431f
I20260927 06:59:27.170910 8410 ts_tablet_manager.cc:1939] T 8bb39f9a944941f4949fe3e446b1431f P 0ec3ef075ade48bbad31c027028e794b: Deleting consensus metadata
I20260927 06:59:27.171218 7911 catalog_manager.cc:4841] TS 0ec3ef075ade48bbad31c027028e794b (127.3.57.3:46543): tablet 8bb39f9a944941f4949fe3e446b1431f (table test-table [id=86163fb68f454d83a75f6e8b65472df7]) successfully deleted
I20260927 06:59:27.174335 8412 tablet_replica.cc:330] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c: stopping tablet replica
I20260927 06:59:27.174453 8412 raft_consensus.cc:2229] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 06:59:27.174436 7909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:35465 (error 108)
I20260927 06:59:27.174561 8412 raft_consensus.cc:2258] T 8bb39f9a944941f4949fe3e446b1431f P 76042b862ffe447cbcd4d5be33fb187c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:27.174669 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8115
I20260927 06:59:27.178973 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8246
I20260927 06:59:27.179000 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.178839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:35465 (local address 127.0.0.1:41122)
0927 06:59:27.178887 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.178953 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:35465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 06:59:27.179104 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.178954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34373 (local address 127.0.0.1:51414)
0927 06:59:27.179044 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.179072 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 06:59:27.179684 7911 catalog_manager.cc:4584] TS 76042b862ffe447cbcd4d5be33fb187c (127.3.57.2:34373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bb39f9a944941f4949fe3e446b1431f: Network error: Client connection negotiation failed: client connection to 127.3.57.2:34373: connect: Connection refused (error 111)
I20260927 06:59:27.183485 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 7893
2026-09-27T06:59:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (623 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 06:59:27.202805 3300 mini_oidc.cc:152] Starting JWKS server
I20260927 06:59:27.204032 3300 webserver.cc:460] Webserver started at https://127.0.0.1:33527/ using document root <none> and password file <none>
I20260927 06:59:27.204133 3300 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:59:27.204304 3300 webserver.cc:460] Webserver started at http://127.0.0.1:40143/ using document root <none> and password file <none>
2026-09-27T06:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:27Z Disabled control of system clock
I20260927 06:59:27.211530 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34753
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:35365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:34753
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/testchainca.cert with env {}
W20260927 06:59:27.289304 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:27.289486 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:27.289507 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:27.290603 8419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:27.290637 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:27.290648 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:27.290661 8419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:27.290673 8419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:59:27.290685 8419 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:59:27.290705 8419 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33527/jwks.json
I20260927 06:59:27.292119 8419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:34753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:27.292297 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:27.292443 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:27.296914 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.297014 8419 server_base.cc:1034] running on GCE node
W20260927 06:59:27.296924 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.296922 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.297374 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:27.297641 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:27.298767 8419 hybrid_clock.cc:648] HybridClock initialized: now 1790492367298760 us; error 31 us; skew 500 ppm
I20260927 06:59:27.299924 8419 webserver.cc:460] Webserver started at http://127.3.57.62:33129/ using document root <none> and password file <none>
I20260927 06:59:27.300135 8419 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:27.300177 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:27.300354 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:27.301311 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "5553a2a73ffb4711a6ba138ae7b57c5d"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "4959bd12769fef7759ee5797efe84c39"
server_key_iv: "dc414d8f6bfad94d90f5f7d3c08f80b0"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.301644 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "5553a2a73ffb4711a6ba138ae7b57c5d"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "4959bd12769fef7759ee5797efe84c39"
server_key_iv: "dc414d8f6bfad94d90f5f7d3c08f80b0"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.302870 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 06:59:27.303540 8433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.303764 8419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.303864 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "5553a2a73ffb4711a6ba138ae7b57c5d"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "4959bd12769fef7759ee5797efe84c39"
server_key_iv: "dc414d8f6bfad94d90f5f7d3c08f80b0"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.303943 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:27.324343 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:27.324616 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:27.324753 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:27.328559 8419 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:34753
I20260927 06:59:27.328593 8485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:34753 every 8 connection(s)
I20260927 06:59:27.328903 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:27.329768 8486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.332021 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d: Bootstrap starting.
I20260927 06:59:27.332595 8486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.332880 8486 log.cc:824] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.333513 8486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d: No bootstrap required, opened a new log
I20260927 06:59:27.335011 8486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } }
I20260927 06:59:27.335163 8486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.335201 8486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5553a2a73ffb4711a6ba138ae7b57c5d, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.335295 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8419
I20260927 06:59:27.335403 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:27.335374 8486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } }
I20260927 06:59:27.335472 8486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:27.335513 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:27.335590 8486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.335623 3300 external_mini_cluster.cc:1428] Setting key 637397385cb5c55d73c47dbdc5c26613
I20260927 06:59:27.336299 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } }
I20260927 06:59:27.336404 8486 leader_election.cc:302] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5553a2a73ffb4711a6ba138ae7b57c5d; no voters:
I20260927 06:59:27.336565 8486 leader_election.cc:288] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:27.336616 8491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:27.336755 8491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [term 1 LEADER]: Becoming Leader. State: Replica: 5553a2a73ffb4711a6ba138ae7b57c5d, State: Running, Role: LEADER
I20260927 06:59:27.336797 8486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:27.336834 8491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } }
I20260927 06:59:27.337203 8492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } } }
I20260927 06:59:27.337288 8492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.337692 8498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:27.337669 8493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5553a2a73ffb4711a6ba138ae7b57c5d. Latest consensus state: current_term: 1 leader_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5553a2a73ffb4711a6ba138ae7b57c5d" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34753 } } }
I20260927 06:59:27.337898 8493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.338173 8498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:27.339771 8498 catalog_manager.cc:1347] Generated new cluster ID: f2c0289cebc748128a02ee2ac3ca7f16
I20260927 06:59:27.339830 8498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:27.341964 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.336064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.3.57.62:34753)
0927 06:59:27.336291 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.336300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.336335 (+ 35us) server_negotiation.cc:407] Connection header received
0927 06:59:27.336457 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.336464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.336537 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.336679 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.337049 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.338868 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.339953 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.340558 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.340664 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.341340 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.341344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.341344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.341526 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.341562 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.341573 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.341627 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.341708 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.341777 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260927 06:59:27.342743 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:27.354199 8498 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:27.354861 8498 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:27.359784 8498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5553a2a73ffb4711a6ba138ae7b57c5d: Generated new TSK 0
I20260927 06:59:27.359941 8498 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 06:59:27.379839 8516 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 06:59:27.379926 8516 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 06:59:27.380196 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.365046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.3.57.62:34753)
0927 06:59:27.365092 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.365094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.365228 (+ 134us) server_negotiation.cc:407] Connection header received
0927 06:59:27.365283 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.365283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.365301 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.365380 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:59:27.365496 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.366258 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.367017 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.367326 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.367395 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.367578 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:59:27.379984 (+ 12406us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 06:59:27.380121 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 06:59:27.380249 8490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 06:59:27.380529 8515 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:27.364979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34753 (local address 127.0.0.1:49744)
0927 06:59:27.365157 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.365166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.365219 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.365380 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.365381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.365382 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:59:27.365437 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.365439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.366366 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.366367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.366938 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.366939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.367392 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.367394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.367436 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.367451 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:59:27.380120 (+ 12669us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 06:59:27.380215 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.57.62:34753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.57.62:34753: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33527/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 06:59:27.380934 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8419
2026-09-27T06:59:27Z chronyd exiting
I20260927 06:59:27.416712 3300 test_util.cc:175] -----------------------------------------------
I20260927 06:59:27.416797 3300 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492341973781-3300-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 06:59:27.417564 3300 mini_oidc.cc:152] Starting JWKS server
I20260927 06:59:27.418268 3300 webserver.cc:460] Webserver started at https://127.0.0.1:42179/ using document root <none> and password file <none>
I20260927 06:59:27.418308 3300 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:59:27.418502 3300 webserver.cc:460] Webserver started at http://127.0.0.1:33929/ using document root <none> and password file <none>
2026-09-27T06:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:27Z Disabled control of system clock
I20260927 06:59:27.425879 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45601
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:45601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42179/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/testchainca.cert with env {}
W20260927 06:59:27.504745 8522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:27.504926 8522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:27.504973 8522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:27.506059 8522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:27.506093 8522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:27.506105 8522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:27.506117 8522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:27.506129 8522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:59:27.506141 8522 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:59:27.506157 8522 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42179/jwks.json
I20260927 06:59:27.507599 8522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:45601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42179/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:27.507797 8522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:27.507944 8522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:27.512410 8528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.512436 8530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.512431 8527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.512579 8522 server_base.cc:1034] running on GCE node
I20260927 06:59:27.512812 8522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:27.513094 8522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:27.514237 8522 hybrid_clock.cc:648] HybridClock initialized: now 1790492367514225 us; error 32 us; skew 500 ppm
I20260927 06:59:27.515442 8522 webserver.cc:460] Webserver started at http://127.3.57.62:38093/ using document root <none> and password file <none>
I20260927 06:59:27.515698 8522 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:27.515748 8522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:27.515867 8522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:27.516826 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "5881e6986d324a36863bf5d97e60b4f5"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "1d94ecbe8a5b5490ad87794afe4a1350"
server_key_iv: "84f8f79ebf2a796e075d74c4980ef13e"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.517160 8522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "5881e6986d324a36863bf5d97e60b4f5"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "1d94ecbe8a5b5490ad87794afe4a1350"
server_key_iv: "84f8f79ebf2a796e075d74c4980ef13e"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.518386 8522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:27.519044 8536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.519214 8522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:27.519336 8522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "5881e6986d324a36863bf5d97e60b4f5"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "1d94ecbe8a5b5490ad87794afe4a1350"
server_key_iv: "84f8f79ebf2a796e075d74c4980ef13e"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.519413 8522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:27.548017 8522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:27.548331 8522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:27.548483 8522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:27.552157 8522 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:45601
I20260927 06:59:27.552187 8588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:45601 every 8 connection(s)
I20260927 06:59:27.552510 8522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:27.553406 8589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.555619 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5: Bootstrap starting.
I20260927 06:59:27.556172 8589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.556444 8589 log.cc:824] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.557073 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5: No bootstrap required, opened a new log
I20260927 06:59:27.558540 8589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } }
I20260927 06:59:27.558665 8589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.558708 8589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5881e6986d324a36863bf5d97e60b4f5, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.558863 8589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } }
I20260927 06:59:27.558939 8589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:27.558991 8589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:27.559042 8589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.559527 8589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } }
I20260927 06:59:27.559664 8589 leader_election.cc:302] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5881e6986d324a36863bf5d97e60b4f5; no voters:
I20260927 06:59:27.559831 8589 leader_election.cc:288] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:27.559861 8592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:27.559965 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8522
I20260927 06:59:27.560009 8592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [term 1 LEADER]: Becoming Leader. State: Replica: 5881e6986d324a36863bf5d97e60b4f5, State: Running, Role: LEADER
I20260927 06:59:27.560067 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:27.560111 8589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:27.560122 8592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } }
I20260927 06:59:27.560307 3300 external_mini_cluster.cc:1428] Setting key 37bec694a0717eba87ad5360d460397a
I20260927 06:59:27.560477 8592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5881e6986d324a36863bf5d97e60b4f5. Latest consensus state: current_term: 1 leader_uuid: "5881e6986d324a36863bf5d97e60b4f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } } }
I20260927 06:59:27.560566 8592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.560485 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5881e6986d324a36863bf5d97e60b4f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5881e6986d324a36863bf5d97e60b4f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45601 } } }
I20260927 06:59:27.560762 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.560875 8597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:27.561568 8597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:27.563220 8597 catalog_manager.cc:1347] Generated new cluster ID: 608896ce7e004f14874844431f721f86
I20260927 06:59:27.563283 8597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:27.566318 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.560896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.3.57.62:45601)
0927 06:59:27.561210 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.561218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.561252 (+ 34us) server_negotiation.cc:407] Connection header received
0927 06:59:27.561366 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.561375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.561443 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.561893 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.561980 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.563158 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.564269 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.564836 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.564961 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.565641 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.565645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.565645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.565847 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.565884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.565896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.565953 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.566058 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.566128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":75,"threads_started":1}
I20260927 06:59:27.567167 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:27.573345 8597 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:27.573987 8597 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:27.582458 8597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5881e6986d324a36863bf5d97e60b4f5: Generated new TSK 0
I20260927 06:59:27.582615 8597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 06:59:27.603188 8619 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 06:59:27.603287 8619 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 06:59:27.603606 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.589448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.3.57.62:45601)
0927 06:59:27.589488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.589490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.589550 (+ 60us) server_negotiation.cc:407] Connection header received
0927 06:59:27.589616 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.589617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.589626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.589676 (+ 50us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:59:27.589803 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.590280 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.591395 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.591660 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.591709 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.591875 (+ 166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:59:27.603395 (+ 11520us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 06:59:27.603503 (+ 108us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 06:59:27.603667 8603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 06:59:27.603739 8618 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:27.589343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45601 (local address 127.0.0.1:59436)
0927 06:59:27.589479 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.589490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.589557 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.589685 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.589686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.589687 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:59:27.589746 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.589748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.590356 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.590357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.591304 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.591306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.591720 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.591724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.591790 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.591795 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:59:27.603488 (+ 11693us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 06:59:27.603638 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.57.62:45601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42179/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
I20260927 06:59:27.603977 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8522
2026-09-27T06:59:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 06:59:27.641443 3300 mini_oidc.cc:152] Starting JWKS server
I20260927 06:59:27.642134 3300 webserver.cc:460] Webserver started at https://127.0.0.1:44873/ using document root <none> and password file <none>
I20260927 06:59:27.642174 3300 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 06:59:27.642340 3300 webserver.cc:460] Webserver started at http://127.0.0.1:41555/ using document root <none> and password file <none>
2026-09-27T06:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:27Z Disabled control of system clock
I20260927 06:59:27.649438 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:43623
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:43623
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 06:59:27.728405 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:27.728588 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:27.728608 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:27.729744 8625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:27.729789 8625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:27.729802 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:27.729851 8625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:27.729874 8625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:59:27.729900 8625 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:59:27.729944 8625 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44873/jwks.json
I20260927 06:59:27.731307 8625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:43623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:27.731576 8625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:27.731860 8625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:27.736888 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.737001 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.737170 8625 server_base.cc:1034] running on GCE node
W20260927 06:59:27.736902 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.737542 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:27.737859 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:27.739046 8625 hybrid_clock.cc:648] HybridClock initialized: now 1790492367739015 us; error 46 us; skew 500 ppm
I20260927 06:59:27.740759 8625 webserver.cc:460] Webserver started at http://127.3.57.62:39865/ using document root <none> and password file <none>
I20260927 06:59:27.741014 8625 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:27.741075 8625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:27.741214 8625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:27.742532 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "90beebaa5b174af78d88eaa3fc5839e9"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "43f8594ff6ae8504f5177a123edee85d"
server_key_iv: "86714696abd1d511510ab9b0d264b173"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.742933 8625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "90beebaa5b174af78d88eaa3fc5839e9"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "43f8594ff6ae8504f5177a123edee85d"
server_key_iv: "86714696abd1d511510ab9b0d264b173"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.744463 8625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:27.745321 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.745487 8625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:27.745580 8625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "90beebaa5b174af78d88eaa3fc5839e9"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "43f8594ff6ae8504f5177a123edee85d"
server_key_iv: "86714696abd1d511510ab9b0d264b173"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.745661 8625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:27.775336 8625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:27.775631 8625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:27.775779 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:27.779515 8625 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:43623
I20260927 06:59:27.779621 8691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:43623 every 8 connection(s)
I20260927 06:59:27.779914 8625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:27.780491 8692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.782799 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: Bootstrap starting.
I20260927 06:59:27.783313 8692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.783623 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8625
I20260927 06:59:27.783672 8692 log.cc:824] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.783721 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:27.783936 3300 external_mini_cluster.cc:1428] Setting key 69d27365dc84af2edf3d503814f4c277
I20260927 06:59:27.784413 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: No bootstrap required, opened a new log
I20260927 06:59:27.786002 8692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } }
I20260927 06:59:27.786151 8692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:27.786188 8692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90beebaa5b174af78d88eaa3fc5839e9, State: Initialized, Role: FOLLOWER
I20260927 06:59:27.786319 8692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } }
I20260927 06:59:27.786407 8692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:27.786453 8692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:27.786514 8692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:27.787001 8692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } }
I20260927 06:59:27.787081 8692 leader_election.cc:302] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90beebaa5b174af78d88eaa3fc5839e9; no voters:
I20260927 06:59:27.787245 8692 leader_election.cc:288] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:27.787307 8697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:27.787441 8697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [term 1 LEADER]: Becoming Leader. State: Replica: 90beebaa5b174af78d88eaa3fc5839e9, State: Running, Role: LEADER
I20260927 06:59:27.787478 8692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:27.787519 8697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } }
I20260927 06:59:27.787813 8697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90beebaa5b174af78d88eaa3fc5839e9. Latest consensus state: current_term: 1 leader_uuid: "90beebaa5b174af78d88eaa3fc5839e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } } }
I20260927 06:59:27.787882 8697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:27.787822 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90beebaa5b174af78d88eaa3fc5839e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90beebaa5b174af78d88eaa3fc5839e9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 43623 } } }
I20260927 06:59:27.788079 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9 [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:27.789397 8713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:27.789455 8713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:27.789508 8703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:27.789664 8703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:27.791119 8703 catalog_manager.cc:1347] Generated new cluster ID: e75f71783e1140e9a45bddf17b4377e0
I20260927 06:59:27.791173 8703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:27.791517 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.784404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.3.57.62:43623)
0927 06:59:27.784588 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.784595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.784617 (+ 22us) server_negotiation.cc:407] Connection header received
0927 06:59:27.785984 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.785990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.786045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.786172 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:27.786554 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.788119 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.789207 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.789817 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.789940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.790830 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:27.790834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:27.790835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:27.791025 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:27.791075 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:27.791090 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:27.791160 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:27.791250 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 06:59:27.791333 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260927 06:59:27.796311 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:27.825482 8703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:27.826140 8703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:27.829939 8703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90beebaa5b174af78d88eaa3fc5839e9: Generated new TSK 0
I20260927 06:59:27.830094 8703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:27.841701 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:27.838337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.3.57.62:43623)
0927 06:59:27.838404 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:27.838406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:27.838551 (+ 145us) server_negotiation.cc:407] Connection header received
0927 06:59:27.838618 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:27.838619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:27.838629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:27.838688 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:59:27.838817 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.839262 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.840407 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.840575 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.840625 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.840806 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:59:27.841500 (+ 694us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 06:59:27.841632 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 06:59:27.841750 8696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 06:59:27.841812 8721 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:27.838315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:43623 (local address 127.0.0.1:58402)
0927 06:59:27.838475 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:27.838486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:27.838551 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:27.838694 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:27.838695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:27.838696 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:59:27.838755 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.838757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.839357 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.839357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.840332 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:27.840334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:27.840640 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:27.840647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:27.840704 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:27.840709 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:59:27.841601 (+ 892us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 06:59:27.841682 (+ 81us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.57.62:43623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260927 06:59:27.841969 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8625
E20260927 06:59:27.852561 8722 webserver.cc:533] Webserver: sslize failed:
E20260927 06:59:27.852613 8722 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T06:59:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T06:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:27Z Disabled control of system clock
I20260927 06:59:27.886847 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45839
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:45839
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/testchainca.cert with env {}
W20260927 06:59:27.964861 8726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:27.965044 8726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:27.965066 8726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:27.966150 8726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:27.966183 8726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:27.966197 8726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:27.966209 8726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:27.966221 8726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 06:59:27.966233 8726 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 06:59:27.966248 8726 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 06:59:27.967641 8726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:45839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:27.967819 8726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:27.967967 8726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:59:27.972628 8726 server_base.cc:1034] running on GCE node
W20260927 06:59:27.972584 8732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.972654 8734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:27.972656 8731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:27.972990 8726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:27.973260 8726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:27.974396 8726 hybrid_clock.cc:648] HybridClock initialized: now 1790492367974378 us; error 45 us; skew 500 ppm
I20260927 06:59:27.975490 8726 webserver.cc:460] Webserver started at http://127.3.57.62:40773/ using document root <none> and password file <none>
I20260927 06:59:27.975747 8726 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:27.975795 8726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:27.975956 8726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:27.976910 8726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "f8e29589435b3522e47363d0355577ae"
server_key_iv: "e10e518731a013a56e1b34fc6f5dd9f8"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.977291 8726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "f8e29589435b3522e47363d0355577ae"
server_key_iv: "e10e518731a013a56e1b34fc6f5dd9f8"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.978444 8726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:27.979099 8740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:27.979259 8726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:27.979347 8726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea"
format_stamp: "Formatted at 2026-09-27 06:59:27 on dist-test-slave-mk3h"
server_key: "f8e29589435b3522e47363d0355577ae"
server_key_iv: "e10e518731a013a56e1b34fc6f5dd9f8"
server_key_version: "encryptionkey@0"
I20260927 06:59:27.979414 8726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:27.989501 8726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:27.989753 8726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:27.989885 8726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:27.993551 8726 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:45839
I20260927 06:59:27.993609 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:45839 every 8 connection(s)
I20260927 06:59:27.993896 8726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:27.994772 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:27.997030 8793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea: Bootstrap starting.
I20260927 06:59:27.997511 8793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:27.997737 8793 log.cc:824] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:27.998315 8793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea: No bootstrap required, opened a new log
I20260927 06:59:27.999840 8793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } }
I20260927 06:59:27.999969 8793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:28.000003 8793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f9fc368f5e425ba00fa6c1b2bd24ea, State: Initialized, Role: FOLLOWER
I20260927 06:59:28.000119 8793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } }
I20260927 06:59:28.000167 8793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:28.000186 8793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:28.000211 8793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:28.000550 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8726
I20260927 06:59:28.000660 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:28.000720 8793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } }
I20260927 06:59:28.000840 8793 leader_election.cc:302] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f9fc368f5e425ba00fa6c1b2bd24ea; no voters:
I20260927 06:59:28.000866 3300 external_mini_cluster.cc:1428] Setting key d2c8bfa369711f08ce5949fa1f7f5d84
I20260927 06:59:28.001024 8793 leader_election.cc:288] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:28.001065 8796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:28.001181 8796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [term 1 LEADER]: Becoming Leader. State: Replica: 68f9fc368f5e425ba00fa6c1b2bd24ea, State: Running, Role: LEADER
I20260927 06:59:28.001278 8793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:28.001267 8796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } }
I20260927 06:59:28.001569 8796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f9fc368f5e425ba00fa6c1b2bd24ea. Latest consensus state: current_term: 1 leader_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } } }
I20260927 06:59:28.001641 8796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:28.001664 8798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f9fc368f5e425ba00fa6c1b2bd24ea" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45839 } } }
I20260927 06:59:28.001748 8798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:28.001961 8803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:28.002450 8803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:28.004894 8803 catalog_manager.cc:1347] Generated new cluster ID: 87a2037bd8264a409c4347fb5c264c5c
I20260927 06:59:28.004964 8803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:28.007416 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:28.001446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.3.57.62:45839)
0927 06:59:28.001892 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:28.001901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:28.002061 (+ 160us) server_negotiation.cc:407] Connection header received
0927 06:59:28.002263 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:28.002271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:28.002338 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:28.002477 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:28.002873 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.004244 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.005253 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.005852 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.005994 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:28.006652 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:28.006656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:28.006656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 06:59:28.006860 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:28.006899 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 06:59:28.006913 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:28.006992 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:28.007056 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 06:59:28.007132 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":96,"threads_started":1}
I20260927 06:59:28.012431 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 06:59:28.027709 8803 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:28.028343 8803 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:28.036666 8803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f9fc368f5e425ba00fa6c1b2bd24ea: Generated new TSK 0
I20260927 06:59:28.036828 8803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 06:59:28.568102 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:28.050425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.3.57.62:45839)
0927 06:59:28.050466 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:28.050468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:28.050574 (+ 106us) server_negotiation.cc:407] Connection header received
0927 06:59:28.050623 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:28.050624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:28.050636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:28.050713 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0927 06:59:28.050824 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.051606 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.052348 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.052647 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.052715 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:28.052854 (+ 139us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 06:59:28.567856 (+515002us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 06:59:28.568013 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 06:59:28.568199 8801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 06:59:28.568202 8822 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:28.050362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45839 (local address 127.0.0.1:45198)
0927 06:59:28.050502 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:28.050513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:28.050571 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:28.050717 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:28.050718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:28.050719 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 06:59:28.050767 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:28.050769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.051691 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.051692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:28.052270 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:28.052272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:28.052719 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:28.052725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:28.052767 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:28.052771 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 06:59:28.567991 (+515220us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 06:59:28.568126 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.57.62:45839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260927 06:59:28.568602 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8726
2026-09-27T06:59:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (706 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:28 dist-test-slave-mk3h krb5kdc[8834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:28 dist-test-slave-mk3h krb5kdc[8834](info): set up 2 sockets
Sep 27 06:59:28 dist-test-slave-mk3h krb5kdc[8834](info): commencing operation
krb5kdc: starting...
W20260927 06:59:29.779759 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:29 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 06:59:29.890558 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44163
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:29.968714 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:29.968948 8847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:29.968984 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:29.970113 8847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:29.970173 8847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:29.970198 8847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:29.970225 8847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:29.970268 8847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:29.971679 8847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:29.971941 8847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:29.972149 8847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:29.976732 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:29.976811 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:29.976872 8847 server_base.cc:1034] running on GCE node
W20260927 06:59:29.976929 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:29.977280 8847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:29.977577 8847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:29.978742 8847 hybrid_clock.cc:648] HybridClock initialized: now 1790492369978732 us; error 36 us; skew 500 ppm
Sep 27 06:59:29 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492369, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:29.985299 8847 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 06:59:29.985827 8847 webserver.cc:460] Webserver started at http://127.3.57.62:38219/ using document root <none> and password file <none>
I20260927 06:59:29.986059 8847 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:29.986123 8847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:29.986274 8847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:29.987389 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "6b3da893c62f41f9b03ba314f079a3ce"
format_stamp: "Formatted at 2026-09-27 06:59:29 on dist-test-slave-mk3h"
server_key: "3f8de0f4b2d381bdacbc60e70f6a175a"
server_key_iv: "7710b77a1b4072605ec881a6e2d8a1aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:29.987789 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "6b3da893c62f41f9b03ba314f079a3ce"
format_stamp: "Formatted at 2026-09-27 06:59:29 on dist-test-slave-mk3h"
server_key: "3f8de0f4b2d381bdacbc60e70f6a175a"
server_key_iv: "7710b77a1b4072605ec881a6e2d8a1aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:29.989179 8847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:29.989867 8862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:29.990036 8847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:29.990134 8847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "6b3da893c62f41f9b03ba314f079a3ce"
format_stamp: "Formatted at 2026-09-27 06:59:29 on dist-test-slave-mk3h"
server_key: "3f8de0f4b2d381bdacbc60e70f6a175a"
server_key_iv: "7710b77a1b4072605ec881a6e2d8a1aa"
server_key_version: "encryptionkey@0"
I20260927 06:59:29.990204 8847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:30.007233 8847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:30.009099 8847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:30.009254 8847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:30.013307 8847 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:44163
I20260927 06:59:30.013341 8914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:44163 every 8 connection(s)
I20260927 06:59:30.013682 8847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:30.014493 8915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:30.014892 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8847
I20260927 06:59:30.015008 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:30.015193 3300 external_mini_cluster.cc:1428] Setting key 15a7cade98f9ab9786964acd25403d70
I20260927 06:59:30.017463 8915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: Bootstrap starting.
I20260927 06:59:30.018117 8915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:30.018421 8915 log.cc:824] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:30.019098 8915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: No bootstrap required, opened a new log
I20260927 06:59:30.021410 8915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } }
I20260927 06:59:30.021588 8915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:30.021641 8915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3da893c62f41f9b03ba314f079a3ce, State: Initialized, Role: FOLLOWER
I20260927 06:59:30.021765 8915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } }
I20260927 06:59:30.021845 8915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:30.021888 8915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:30.021938 8915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 0 FOLLOWER]: Advancing to term 1
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:30.022521 8915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } }
I20260927 06:59:30.022639 8915 leader_election.cc:302] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b3da893c62f41f9b03ba314f079a3ce; no voters:
I20260927 06:59:30.022797 8915 leader_election.cc:288] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:30.022873 8920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:30.023037 8920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [term 1 LEADER]: Becoming Leader. State: Replica: 6b3da893c62f41f9b03ba314f079a3ce, State: Running, Role: LEADER
I20260927 06:59:30.023222 8920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } }
I20260927 06:59:30.023070 8915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:30.023622 8922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b3da893c62f41f9b03ba314f079a3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } } }
I20260927 06:59:30.023696 8922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:30.023756 8921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3da893c62f41f9b03ba314f079a3ce. Latest consensus state: current_term: 1 leader_uuid: "6b3da893c62f41f9b03ba314f079a3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3da893c62f41f9b03ba314f079a3ce" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44163 } } }
I20260927 06:59:30.023837 8921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce [sys.catalog]: This master's current role is: LEADER
W20260927 06:59:30.025243 8935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:30.025547 8935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:30.025656 8924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:30.025862 8924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:30.026710 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.015774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.3.57.62:44163)
0927 06:59:30.015981 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.015991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.016029 (+ 38us) server_negotiation.cc:407] Connection header received
0927 06:59:30.016533 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.016542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.016601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.016756 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.017349 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.018716 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.019504 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.020141 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.020257 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.022629 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.022632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.022633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.022644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.024914 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.025334 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.025337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.025337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.025563 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.025786 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.025787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.025788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.026091 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.026244 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.026392 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.026524 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260927 06:59:30.027580 8924 catalog_manager.cc:1347] Generated new cluster ID: 48ca629d71064c1ca34c36dd4e30da43
I20260927 06:59:30.027626 8924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 06:59:30.056506 8924 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:30.057058 8924 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:30.065101 8924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b3da893c62f41f9b03ba314f079a3ce: Generated new TSK 0
I20260927 06:59:30.065300 8924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 06:59:30.090085 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:30.168658 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:30.168846 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:30.168869 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:30.169998 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:30.170042 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:30.170068 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:30.171487 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:30.171702 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:30.171875 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 06:59:30.176580 8941 server_base.cc:1034] running on GCE node
W20260927 06:59:30.176564 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.176693 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.176566 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:30.177078 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:30.177326 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:30.178478 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790492370178460 us; error 34 us; skew 500 ppm
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.180800 8941 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 06:59:30.181241 8941 webserver.cc:460] Webserver started at http://127.3.57.1:33769/ using document root <none> and password file <none>
I20260927 06:59:30.181448 8941 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:30.181510 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:30.181644 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:30.182595 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "38ff048fd1ae47e992e61d71140ca077"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "8e7f115fd313b16e49161b0252cd4e5b"
server_key_iv: "1d81596c5a6b487e014c17789ef7fe61"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.182940 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "38ff048fd1ae47e992e61d71140ca077"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "8e7f115fd313b16e49161b0252cd4e5b"
server_key_iv: "1d81596c5a6b487e014c17789ef7fe61"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.184293 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 06:59:30.184988 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.185173 8941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:30.185266 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "38ff048fd1ae47e992e61d71140ca077"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "8e7f115fd313b16e49161b0252cd4e5b"
server_key_iv: "1d81596c5a6b487e014c17789ef7fe61"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.185338 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:30.196610 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:30.198525 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:30.198680 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:30.198894 8941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:30.199200 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:30.199249 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.199288 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:30.199316 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.205374 8941 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:42241
I20260927 06:59:30.205423 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:42241 every 8 connection(s)
I20260927 06:59:30.205739 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:30.214116 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.206414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:48113 (local address 127.3.57.62:44163)
0927 06:59:30.206489 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.206491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.206650 (+ 159us) server_negotiation.cc:407] Connection header received
0927 06:59:30.207542 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.207543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.207582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.207647 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.208193 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.208686 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.209600 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.209794 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.209847 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.211823 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.211824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.211825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.211831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.213001 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.213378 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.213378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.213379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.213427 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.213623 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.213623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.213624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.213721 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.213770 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.213983 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.214051 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 06:59:30.214277 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.206372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44163 (local address 127.3.57.1:48113)
0927 06:59:30.206536 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:30.206571 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:30.207449 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:30.207698 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:30.207701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:30.207942 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:30.208123 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.208126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.208760 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.208760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.209531 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.209533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.209862 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.209862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.209953 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.210379 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:30.210387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:30.211742 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:30.213079 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.213086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.213088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.213307 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.213481 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.213482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.213482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.213574 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.213770 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:30.213774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:30.213882 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 06:59:30.213962 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 06:59:30.214080 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260927 06:59:30.214643 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 8941
I20260927 06:59:30.214766 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:30.214798 9070 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44163
I20260927 06:59:30.214895 9070 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:30.214968 3300 external_mini_cluster.cc:1428] Setting key a4553b75f9399b44633c312878e76471
I20260927 06:59:30.215090 9070 heartbeater.cc:511] Master 127.3.57.62:44163 requested a full tablet report, sending...
I20260927 06:59:30.215641 8878 ts_manager.cc:194] Registered new tserver with Master: 38ff048fd1ae47e992e61d71140ca077 (127.3.57.1:42241)
I20260927 06:59:30.216239 8878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:48113
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 06:59:30.238986 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:30.317817 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:30.318017 9075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:30.318040 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:30.319190 9075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:30.319232 9075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:30.319283 9075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:30.320804 9075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:30.321043 9075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:30.321272 9075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:30.326421 9080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.326462 9083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:30.326436 9075 server_base.cc:1034] running on GCE node
W20260927 06:59:30.326421 9081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:30.326833 9075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:30.327111 9075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:30.328261 9075 hybrid_clock.cc:648] HybridClock initialized: now 1790492370328251 us; error 30 us; skew 500 ppm
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.330587 9075 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 06:59:30.331048 9075 webserver.cc:460] Webserver started at http://127.3.57.2:44957/ using document root <none> and password file <none>
I20260927 06:59:30.331264 9075 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:30.331326 9075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:30.331459 9075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:30.332504 9075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "0560cbef7ffd4dd19e23befa144c2c80"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "cc28f120d64e5a2c65302a4326bd3bd5"
server_key_iv: "8d764ffd9ae510bb15710eef599bc77b"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.332861 9075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "0560cbef7ffd4dd19e23befa144c2c80"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "cc28f120d64e5a2c65302a4326bd3bd5"
server_key_iv: "8d764ffd9ae510bb15710eef599bc77b"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.334177 9075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:30.334898 9090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.335078 9075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:30.335179 9075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "0560cbef7ffd4dd19e23befa144c2c80"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "cc28f120d64e5a2c65302a4326bd3bd5"
server_key_iv: "8d764ffd9ae510bb15710eef599bc77b"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.335258 9075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:30.342561 9075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:30.344292 9075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:30.344444 9075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:30.344641 9075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:30.344910 9075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:30.344966 9075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.345003 9075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:30.345027 9075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.350606 9075 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46197
I20260927 06:59:30.350643 9203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46197 every 8 connection(s)
I20260927 06:59:30.350945 9075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:30.353505 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9075
I20260927 06:59:30.353621 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:30.353802 3300 external_mini_cluster.cc:1428] Setting key e602db0afc6470064f1a00690c9711ff
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:30.360455 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.351634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:54041 (local address 127.3.57.62:44163)
0927 06:59:30.351699 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.351701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.351885 (+ 184us) server_negotiation.cc:407] Connection header received
0927 06:59:30.352860 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.352861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.352875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.352925 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.353522 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.354139 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.355105 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.355309 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.355392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.357689 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.357690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.357691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.357698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.359065 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.359466 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.359468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.359468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.359526 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.359796 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.359797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.359797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.359913 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.359989 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.360257 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.360353 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:59:30.360625 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.351621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44163 (local address 127.3.57.2:54041)
0927 06:59:30.351763 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:30.351803 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:30.352734 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:30.352995 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:30.352999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:30.353242 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:30.353414 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.353418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.354224 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.354225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.355012 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.355014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.355400 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.355401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.355530 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.356107 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:30.356118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:30.357598 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:30.359162 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.359169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.359171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.359388 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.359623 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.359624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.359625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.359730 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.359978 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:30.359985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:30.360119 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 06:59:30.360235 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 06:59:30.360400 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260927 06:59:30.361092 9204 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44163
I20260927 06:59:30.361168 9204 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:30.361352 9204 heartbeater.cc:511] Master 127.3.57.62:44163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 06:59:30.361771 8878 ts_manager.cc:194] Registered new tserver with Master: 0560cbef7ffd4dd19e23befa144c2c80 (127.3.57.2:46197)
I20260927 06:59:30.362164 8878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:54041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 06:59:30.378793 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:30.459623 9209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:30.459818 9209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:30.459841 9209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:30.460935 9209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:30.460968 9209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:30.460994 9209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:30.462344 9209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:30.462522 9209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:30.462742 9209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:30.467478 9215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.467521 9217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:30.467507 9214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:30.467572 9209 server_base.cc:1034] running on GCE node
I20260927 06:59:30.467870 9209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:30.468130 9209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:30.469280 9209 hybrid_clock.cc:648] HybridClock initialized: now 1790492370469266 us; error 30 us; skew 500 ppm
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.471791 9209 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 06:59:30.472241 9209 webserver.cc:460] Webserver started at http://127.3.57.3:35069/ using document root <none> and password file <none>
I20260927 06:59:30.472466 9209 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:30.472510 9209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:30.472658 9209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:30.473678 9209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "128445842d9d4d34886d1555762bf529"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "b5cbbbe759e57b92bcfda49ad39f584e"
server_key_iv: "cde0042db093b36808a2be8e5b51137a"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.474031 9209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "128445842d9d4d34886d1555762bf529"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "b5cbbbe759e57b92bcfda49ad39f584e"
server_key_iv: "cde0042db093b36808a2be8e5b51137a"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.475263 9209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:30.476009 9224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.476176 9209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.476267 9209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "128445842d9d4d34886d1555762bf529"
format_stamp: "Formatted at 2026-09-27 06:59:30 on dist-test-slave-mk3h"
server_key: "b5cbbbe759e57b92bcfda49ad39f584e"
server_key_iv: "cde0042db093b36808a2be8e5b51137a"
server_key_version: "encryptionkey@0"
I20260927 06:59:30.476348 9209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:30.486594 9209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:30.488406 9209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:30.488554 9209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:30.488758 9209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:30.489048 9209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:30.489095 9209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.489130 9209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:30.489156 9209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:30.495000 9209 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:42181
I20260927 06:59:30.495039 9337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:42181 every 8 connection(s)
I20260927 06:59:30.495347 9209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 06:59:30.503268 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9209
I20260927 06:59:30.503386 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:30.503651 3300 external_mini_cluster.cc:1428] Setting key 9fe191cd73cf51b896d78eb0f9b57264
I20260927 06:59:30.504518 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.496088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:51223 (local address 127.3.57.62:44163)
0927 06:59:30.496155 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.496157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.496334 (+ 177us) server_negotiation.cc:407] Connection header received
0927 06:59:30.497237 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.497238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.497246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.497303 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.497841 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.498334 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.499336 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.499519 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.499628 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.501709 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.501711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.501711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.501717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.503021 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.503509 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.503510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.503510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.503603 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.503887 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.503889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.503889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.504013 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.504084 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.504345 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.504421 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 06:59:30.504702 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.496073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44163 (local address 127.3.57.3:51223)
0927 06:59:30.496214 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:30.496255 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:30.497153 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:30.497354 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:30.497357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:30.497598 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:30.497776 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.497779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.498439 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.498440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.499252 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:30.499254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.499658 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.499660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:30.499767 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.500220 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:30.500227 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:30.501619 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:30.503110 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.503118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.503120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.503428 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.503670 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:30.503671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:30.503672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:30.503822 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:30.504088 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:30.504095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:30.504222 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 06:59:30.504325 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:59:30.504488 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260927 06:59:30.505069 9338 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44163
I20260927 06:59:30.505169 9338 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:30.505333 9338 heartbeater.cc:511] Master 127.3.57.62:44163 requested a full tablet report, sending...
I20260927 06:59:30.505671 8878 ts_manager.cc:194] Registered new tserver with Master: 128445842d9d4d34886d1555762bf529 (127.3.57.3:42181)
I20260927 06:59:30.506032 8878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:51223
I20260927 06:59:30.515211 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/test-admin.keytab.
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.529791 3300 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 06:59:30.531229 9345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:30.530560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.3.57.1:42241)
0927 06:59:30.530746 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.530751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.530761 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:30.530786 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.530788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.530801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.530875 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.531150 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38652: BlockingRecv error: recv got EOF from 127.0.0.1:38652 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260927 06:59:30.531348 3300 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc' of type FILE
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.532284 3300 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.533108 3300 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:30.540110 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.533611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.3.57.1:42241)
0927 06:59:30.533651 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.533654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.533666 (+ 12us) server_negotiation.cc:407] Connection header received
0927 06:59:30.533716 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.533716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.533725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.533771 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.534113 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.534804 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.535450 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.535807 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.535857 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.537727 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.537734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.537734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.537740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.539156 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.539454 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.539456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.539457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.539505 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.539708 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.539709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.539709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.539831 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.539925 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.539989 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.540044 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 06:59:30.541757 9345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:30.541215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.3.57.1:42241)
0927 06:59:30.541271 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.541273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.541318 (+ 45us) server_negotiation.cc:407] Connection header received
0927 06:59:30.541386 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.541387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.541396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.541450 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.541711 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38668: BlockingRecv error: recv got EOF from 127.0.0.1:38668 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 06:59:30.541816 3300 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc' of type FILE
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.542665 3300 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.543460 3300 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:30.549794 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.543922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.3.57.1:42241)
0927 06:59:30.543962 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.543963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.543995 (+ 32us) server_negotiation.cc:407] Connection header received
0927 06:59:30.544064 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.544065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.544075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.544135 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.544489 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.544946 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.545603 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.545792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.545871 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.547684 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.547686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.547686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.547693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.548779 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.549101 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.549102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.549102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.549151 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.549337 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.549338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.549338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.549432 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.549530 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.549639 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.549709 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 06:59:30.551041 9345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 06:59:30.550638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.3.57.1:42241)
0927 06:59:30.550671 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.550673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.550696 (+ 23us) server_negotiation.cc:407] Connection header received
0927 06:59:30.550742 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.550743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.550749 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.550790 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.551002 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38684: BlockingRecv error: recv got EOF from 127.0.0.1:38684 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 06:59:30.551146 3300 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492341973781-3300-0/krb5kdc/krb5cc' of type FILE
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.552073 3300 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:30.552870 3300 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[8834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 06:59:30.558867 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:30.553316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.3.57.1:42241)
0927 06:59:30.553350 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:30.553352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:30.553387 (+ 35us) server_negotiation.cc:407] Connection header received
0927 06:59:30.553454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:30.553455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:30.553461 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:30.553497 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:30.553796 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.554224 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.554836 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:30.555001 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:30.555047 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:30.556867 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:30.556868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:30.556869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:30.556874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:30.558110 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.558371 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.558371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.558372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.558415 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:30.558559 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:30.558560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:30.558560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:30.558629 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:30.558692 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:30.558772 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 06:59:30.558819 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 06:59:30.559392 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8941
I20260927 06:59:30.565212 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9075
I20260927 06:59:30.569331 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9209
I20260927 06:59:30.573617 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 8847
2026-09-27T06:59:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1821 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[9358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[9358](info): set up 2 sockets
Sep 27 06:59:30 dist-test-slave-mk3h krb5kdc[9358](info): commencing operation
krb5kdc: starting...
W20260927 06:59:31.599777 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 06:59:31 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T06:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T06:59:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab.
Entry for principal oryx/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab.
I20260927 06:59:31.710664 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41287
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab
--principal=oryx/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:31.795467 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:31.795698 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:31.795722 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:31.796859 9371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:31.796907 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:31.796921 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:31.796934 9371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:31.796947 9371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:31.798427 9371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab
--principal=oryx/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:31.798671 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:31.798903 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:31.803627 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:31.803700 9371 server_base.cc:1034] running on GCE node
W20260927 06:59:31.803710 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:31.803627 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:31.804113 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:31.804437 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:31.805593 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790492371805591 us; error 37 us; skew 500 ppm
Sep 27 06:59:31 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:31.808025 9371 init.cc:377] Logged in from keytab as oryx/127.3.57.62@KRBTEST.COM (short username oryx)
I20260927 06:59:31.808480 9371 webserver.cc:460] Webserver started at http://127.3.57.62:37883/ using document root <none> and password file <none>
I20260927 06:59:31.808717 9371 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:31.808781 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:31.808919 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:31.809937 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "57f2e6c4243c434d9b1d25ff9653e653"
format_stamp: "Formatted at 2026-09-27 06:59:31 on dist-test-slave-mk3h"
server_key: "31a09b93059d6fdfff653d515bb98bb5"
server_key_iv: "e144466b8d0e3b5b6f89b083bfe6e206"
server_key_version: "encryptionkey@0"
I20260927 06:59:31.810297 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "57f2e6c4243c434d9b1d25ff9653e653"
format_stamp: "Formatted at 2026-09-27 06:59:31 on dist-test-slave-mk3h"
server_key: "31a09b93059d6fdfff653d515bb98bb5"
server_key_iv: "e144466b8d0e3b5b6f89b083bfe6e206"
server_key_version: "encryptionkey@0"
I20260927 06:59:31.811586 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 06:59:31.812343 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:31.812518 9371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:31.812611 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "57f2e6c4243c434d9b1d25ff9653e653"
format_stamp: "Formatted at 2026-09-27 06:59:31 on dist-test-slave-mk3h"
server_key: "31a09b93059d6fdfff653d515bb98bb5"
server_key_iv: "e144466b8d0e3b5b6f89b083bfe6e206"
server_key_version: "encryptionkey@0"
I20260927 06:59:31.812691 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:31.831864 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:31.833506 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:31.833649 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:31.837767 9371 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41287
I20260927 06:59:31.837801 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41287 every 8 connection(s)
I20260927 06:59:31.838148 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 06:59:31.839097 9439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:31.841354 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Bootstrap starting.
I20260927 06:59:31.841845 9439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:31.842063 9439 log.cc:824] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:31.842681 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: No bootstrap required, opened a new log
I20260927 06:59:31.844609 9439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } }
I20260927 06:59:31.844755 9439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:31.844784 9439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f2e6c4243c434d9b1d25ff9653e653, State: Initialized, Role: FOLLOWER
I20260927 06:59:31.844930 9439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } }
I20260927 06:59:31.845022 9439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 06:59:31.845055 9439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 06:59:31.845085 9439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:31.845307 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9371
I20260927 06:59:31.845418 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 06:59:31.845623 3300 external_mini_cluster.cc:1428] Setting key 1b8ab1b92fb745f5d54f177b7193a19f
I20260927 06:59:31.846312 9439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } }
I20260927 06:59:31.846427 9439 leader_election.cc:302] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57f2e6c4243c434d9b1d25ff9653e653; no voters:
I20260927 06:59:31.846588 9439 leader_election.cc:288] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 06:59:31.846611 9444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:31.846724 9444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 LEADER]: Becoming Leader. State: Replica: 57f2e6c4243c434d9b1d25ff9653e653, State: Running, Role: LEADER
I20260927 06:59:31.846812 9444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } }
I20260927 06:59:31.846856 9439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:31.847114 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } }
I20260927 06:59:31.847177 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:31.847293 9444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f2e6c4243c434d9b1d25ff9653e653. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } }
I20260927 06:59:31.847352 9444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:31.847481 9448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 06:59:31.848096 9448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 06:59:31.849924 9448 catalog_manager.cc:1347] Generated new cluster ID: 48dab597f0b248e0b7a32b05db3b36d3
I20260927 06:59:31.849982 9448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 06:59:31 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:31.856590 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:31.846148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.3.57.62:41287)
0927 06:59:31.846329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:31.846337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:31.846361 (+ 24us) server_negotiation.cc:407] Connection header received
0927 06:59:31.846473 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:31.846480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:31.846541 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:31.846685 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:31.847373 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:31.848735 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:31.849739 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:31.850330 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:31.850485 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:31.853073 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:31.853078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:31.853078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:31.853091 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:31.854934 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:31.855341 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:31.855345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:31.855345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:31.855451 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:31.855697 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:31.855699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:31.855699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:31.855954 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:31.856140 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:31.856217 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 06:59:31.856378 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260927 06:59:31.862004 9448 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 06:59:31.862572 9448 catalog_manager.cc:1504] Loading token signing keys...
I20260927 06:59:31.868707 9448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Generated new TSK 0
I20260927 06:59:31.868891 9448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
I20260927 06:59:31.900350 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:31.999456 9465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:31.999718 9465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:31.999744 9465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.000919 9465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.000957 9465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.000984 9465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:32.002415 9465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:32.002658 9465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.002885 9465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.007954 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.007954 9473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.007969 9465 server_base.cc:1034] running on GCE node
W20260927 06:59:32.007967 9470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.008374 9465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.008576 9465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.009732 9465 hybrid_clock.cc:648] HybridClock initialized: now 1790492372009718 us; error 25 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.012063 9465 init.cc:377] Logged in from keytab as oryx/127.3.57.1@KRBTEST.COM (short username oryx)
I20260927 06:59:32.012473 9465 webserver.cc:460] Webserver started at http://127.3.57.1:41675/ using document root <none> and password file <none>
I20260927 06:59:32.012691 9465 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.012753 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.012892 9465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.013880 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "da37e7478e5e432a96ee5564f010d2fd"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "2369c194ea69f215f04ce6dd368abbc9"
server_key_iv: "a37cb40a3e727549f1705968350c2940"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.014223 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "da37e7478e5e432a96ee5564f010d2fd"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "2369c194ea69f215f04ce6dd368abbc9"
server_key_iv: "a37cb40a3e727549f1705968350c2940"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.015533 9465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 06:59:32.016274 9480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.016453 9465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:32.016522 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "da37e7478e5e432a96ee5564f010d2fd"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "2369c194ea69f215f04ce6dd368abbc9"
server_key_iv: "a37cb40a3e727549f1705968350c2940"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.016570 9465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.030474 9465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.032240 9465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.032352 9465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.032519 9465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:32.032825 9465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:32.032861 9465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.032912 9465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:32.032936 9465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.038475 9465 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:39683
I20260927 06:59:32.038552 9593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:39683 every 8 connection(s)
I20260927 06:59:32.038998 9465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:32.045280 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9465
I20260927 06:59:32.045383 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 06:59:32.045588 3300 external_mini_cluster.cc:1428] Setting key 0943ebbec043d83fda66ccf71ca091e3
I20260927 06:59:32.048527 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.039738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:56453)
0927 06:59:32.039925 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.039961 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.040431 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.040633 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.040636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.041181 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.041361 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.041365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.042102 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.042103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.042865 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.042866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.043244 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.043245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.043349 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.043869 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.043877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.045248 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.046995 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.047002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.047004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.047360 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.047572 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.047573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.047574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.047715 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.048040 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.048055 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.048173 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.048264 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.048376 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260927 06:59:32.048542 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.039778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:56453 (local address 127.3.57.62:41287)
0927 06:59:32.039859 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.039861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.040054 (+ 193us) server_negotiation.cc:407] Connection header received
0927 06:59:32.040518 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.040519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.040532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.040583 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.041488 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.042002 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.042956 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.043165 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.043224 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.045394 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.045396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.045396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.045404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.046935 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.047427 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.047428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.047428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.047479 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.047801 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.047802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.047802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.047981 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.048049 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.048312 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.048417 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 06:59:32.049111 9594 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:32.049208 9594 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.049393 9594 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
I20260927 06:59:32.049862 9401 ts_manager.cc:194] Registered new tserver with Master: da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683)
I20260927 06:59:32.050498 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:56453
WARNING: no policy specified for oryx/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
I20260927 06:59:32.071372 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:32.151134 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.151360 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.151398 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.152536 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.152596 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.152642 9599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:32.154044 9599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:32.154296 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.154505 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.159163 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.159154 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.159192 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.159430 9599 server_base.cc:1034] running on GCE node
I20260927 06:59:32.159657 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.159904 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.161059 9599 hybrid_clock.cc:648] HybridClock initialized: now 1790492372161040 us; error 32 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.163417 9599 init.cc:377] Logged in from keytab as oryx/127.3.57.2@KRBTEST.COM (short username oryx)
I20260927 06:59:32.163848 9599 webserver.cc:460] Webserver started at http://127.3.57.2:35735/ using document root <none> and password file <none>
I20260927 06:59:32.164031 9599 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.164068 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.164149 9599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.165046 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "45d9609bdbcc443abbea3255ff647e8c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "f235aa41c679bfacf7dbc60987befb7b"
server_key_iv: "780ffcacfd1947d949c36ad24e843eb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.165323 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "45d9609bdbcc443abbea3255ff647e8c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "f235aa41c679bfacf7dbc60987befb7b"
server_key_iv: "780ffcacfd1947d949c36ad24e843eb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.166519 9599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 06:59:32.167210 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.167395 9599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:32.167478 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "45d9609bdbcc443abbea3255ff647e8c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "f235aa41c679bfacf7dbc60987befb7b"
server_key_iv: "780ffcacfd1947d949c36ad24e843eb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.167531 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.184886 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.186703 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.186811 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.186975 9599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:32.187263 9599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:32.187304 9599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.187325 9599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:32.187338 9599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.192901 9599 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:42509
I20260927 06:59:32.192970 9727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:42509 every 8 connection(s)
I20260927 06:59:32.193259 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 06:59:32.195964 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9599
I20260927 06:59:32.196053 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 06:59:32.196274 3300 external_mini_cluster.cc:1428] Setting key d81f806bec539586ddf1ec23ad94d151
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:32.202775 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.193924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:49451 (local address 127.3.57.62:41287)
0927 06:59:32.193987 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.193989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.194191 (+ 202us) server_negotiation.cc:407] Connection header received
0927 06:59:32.194634 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.194635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.194645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.194697 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.195626 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.196233 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.197493 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.197713 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.197785 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.199909 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.199910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.199911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.199918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.201365 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.201804 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.201805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.201805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.201894 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.202147 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.202148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.202149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.202263 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.202336 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.202601 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.202687 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 06:59:32.202968 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.193908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:49451)
0927 06:59:32.194072 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.194105 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.194554 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.194748 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.194751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.195301 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.195483 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.195487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.196343 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.196345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.197392 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.197395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.197791 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.197792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.197926 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.198449 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.198457 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.199819 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.201475 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.201482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.201484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.201730 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.201961 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.201962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.201962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.202075 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.202326 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.202332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.202470 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.202570 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.202725 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260927 06:59:32.203429 9728 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:32.203533 9728 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.203727 9728 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 06:59:32.204106 9401 ts_manager.cc:194] Registered new tserver with Master: 45d9609bdbcc443abbea3255ff647e8c (127.3.57.2:42509)
I20260927 06:59:32.204521 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:49451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
I20260927 06:59:32.220077 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:32.302071 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.302286 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.302323 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.303445 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.303507 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.303548 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:32.305087 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:32.305354 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.305562 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.310281 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.310346 9733 server_base.cc:1034] running on GCE node
W20260927 06:59:32.310220 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.310473 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.310676 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.310909 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.312071 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790492372312055 us; error 30 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.314471 9733 init.cc:377] Logged in from keytab as oryx/127.3.57.3@KRBTEST.COM (short username oryx)
I20260927 06:59:32.314844 9733 webserver.cc:460] Webserver started at http://127.3.57.3:40905/ using document root <none> and password file <none>
I20260927 06:59:32.315008 9733 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.315047 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.315178 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.316231 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "aa73dc4a1b664debaa59feb5eaa7e65b"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b752e740d77759eca68ba1f5fa591d1d"
server_key_iv: "ed868115ced14099c8bacec661d65b0e"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.316562 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "aa73dc4a1b664debaa59feb5eaa7e65b"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b752e740d77759eca68ba1f5fa591d1d"
server_key_iv: "ed868115ced14099c8bacec661d65b0e"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.317754 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 06:59:32.318434 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.318599 9733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:32.318681 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "aa73dc4a1b664debaa59feb5eaa7e65b"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b752e740d77759eca68ba1f5fa591d1d"
server_key_iv: "ed868115ced14099c8bacec661d65b0e"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.318745 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.352052 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.353760 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.353932 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.354151 9733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:32.354439 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:32.354472 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.354494 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:32.354535 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.360347 9733 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:44451
I20260927 06:59:32.360402 9861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:44451 every 8 connection(s)
I20260927 06:59:32.360680 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:32.364545 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9733
I20260927 06:59:32.364629 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 06:59:32.364801 3300 external_mini_cluster.cc:1428] Setting key 9d78cd6afd5d73c68ca18bdfd0733737
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:32.369607 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.361354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:39357 (local address 127.3.57.62:41287)
0927 06:59:32.361414 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.361416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.361610 (+ 194us) server_negotiation.cc:407] Connection header received
0927 06:59:32.362113 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.362115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.362127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.362178 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.363080 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.363640 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.364615 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.364890 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.364961 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.367008 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.367009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.367010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.367017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.368270 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.368662 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.368664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.368664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.368728 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.368978 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.368979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.368980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.369095 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.369183 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.369444 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.369518 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 06:59:32.369807 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.361338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:39357)
0927 06:59:32.361487 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.361528 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.362016 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.362256 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.362260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.362810 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.362986 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.362990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.363728 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.363729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.364512 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.364514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.364967 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.364969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.365089 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.365580 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.365590 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.366921 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.368361 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.368367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.368369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.368590 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.368809 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.368810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.368811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.368913 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.369172 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.369178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.369319 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.369422 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.369568 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260927 06:59:32.370277 9862 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:32.370384 9862 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.370512 9862 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
I20260927 06:59:32.370831 9401 ts_manager.cc:194] Registered new tserver with Master: aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:32.371209 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:39357
I20260927 06:59:32.376360 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.57.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.61.keytab.
Entry for principal oryx/127.3.57.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.61.keytab.
I20260927 06:59:32.398381 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:42163
--webserver_interface=127.3.57.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.61.keytab
--principal=oryx/127.3.57.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41287,127.3.57.61:42163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:32.475965 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.476141 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.476162 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.477255 9867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:32.477294 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.477308 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.477357 9867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:32.477382 9867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:32.478781 9867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41287,127.3.57.61:42163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.61.keytab
--principal=oryx/127.3.57.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.57.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
W20260927 06:59:32.478899 9867 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.57.62:41287,127.3.57.61:42163'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 06:59:32.479029 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.479241 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.483726 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.483765 9867 server_base.cc:1034] running on GCE node
W20260927 06:59:32.483791 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.483736 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.484055 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.484293 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.485436 9867 hybrid_clock.cc:648] HybridClock initialized: now 1790492372485420 us; error 29 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.488245 9867 init.cc:377] Logged in from keytab as oryx/127.3.57.61@KRBTEST.COM (short username oryx)
I20260927 06:59:32.488637 9867 webserver.cc:460] Webserver started at http://127.3.57.61:46661/ using document root <none> and password file <none>
I20260927 06:59:32.488864 9867 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.488929 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.489078 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.490056 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/instance:
uuid: "a555d7cefc394026a632b51a9d409f13"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b16721bc1fb6ab8a4f38d960ef0c0e30"
server_key_iv: "4fb730a562a35da7d34fa56e38a8537b"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.490403 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal/instance:
uuid: "a555d7cefc394026a632b51a9d409f13"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b16721bc1fb6ab8a4f38d960ef0c0e30"
server_key_iv: "4fb730a562a35da7d34fa56e38a8537b"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.491678 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 06:59:32.492415 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.492595 9867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:32.492687 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "a555d7cefc394026a632b51a9d409f13"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b16721bc1fb6ab8a4f38d960ef0c0e30"
server_key_iv: "4fb730a562a35da7d34fa56e38a8537b"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.492751 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.502830 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.504921 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.505075 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.508690 9867 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.61:42163
I20260927 06:59:32.508721 9934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.61:42163 every 8 connection(s)
I20260927 06:59:32.509032 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
I20260927 06:59:32.509922 9935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:32.511309 9935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } has no permanent_uuid. Determining permanent_uuid...
I20260927 06:59:32.512503 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 9867
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.61@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
I20260927 06:59:32.521204 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.511990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.3.57.62:41287)
0927 06:59:32.512046 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.512048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.512235 (+ 187us) server_negotiation.cc:407] Connection header received
0927 06:59:32.512841 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.512842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.512854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.512917 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.513692 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.514494 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.515700 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.515947 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.516017 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.518086 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.518087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.518088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.518095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.519831 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.520291 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.520292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.520292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.520351 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.520586 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.520586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.520587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.520745 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.520810 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.521048 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.521118 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 06:59:32.521473 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.511961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37098)
0927 06:59:32.512120 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.512159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.512745 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.513154 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.513159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.513468 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.513605 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.513610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.514607 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.514608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.515603 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.515606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.516025 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.516026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.516116 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.516590 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.516597 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.518005 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.519900 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.519905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.519907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.520224 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.520414 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.520415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.520416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.520514 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.520801 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.520806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.520931 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.521028 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.521161 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260927 06:59:32.521679 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.512974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.3.57.61:42163)
0927 06:59:32.513116 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.513120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.513155 (+ 35us) server_negotiation.cc:407] Connection header received
0927 06:59:32.513207 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.513210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.513232 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.513311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.513823 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.514884 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.515886 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.516316 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.516397 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.518579 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.518585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.518585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.518592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.520317 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.520623 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.520626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.520626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.520681 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.520858 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.520859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.520860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.520991 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.521136 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.521204 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.521565 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260927 06:59:32.522070 9935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.61@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
I20260927 06:59:32.527325 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.522292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38674)
0927 06:59:32.522328 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.522336 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.522365 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.522493 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.522493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.522585 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.522622 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.522623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.523157 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.523158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.523635 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.523637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.523940 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.523940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.524005 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.524441 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.524445 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.525470 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.526646 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.526647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.526647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.526793 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.526945 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.526945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.526946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.526991 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.527140 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.527141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.527168 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.527212 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.527275 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 06:59:32.527387 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.522324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.3.57.61:42163)
0927 06:59:32.522367 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.522369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.522379 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:32.522413 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.522414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.522423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.522491 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.522664 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.523104 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.523717 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.523889 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.523934 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.525540 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.525541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.525541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.525547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.526584 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.526851 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.526852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.526852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.526895 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.527027 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.527027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.527027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.527101 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.527143 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.527225 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.527316 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":26}
I20260927 06:59:32.528431 9935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Bootstrap starting.
I20260927 06:59:32.529150 9935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:32.529408 9935 log.cc:824] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:32.530014 9935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: No bootstrap required, opened a new log
I20260927 06:59:32.531280 9935 raft_consensus.cc:348] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } }
I20260927 06:59:32.531394 9935 raft_consensus.cc:374] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:32.531416 9935 raft_consensus.cc:729] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a555d7cefc394026a632b51a9d409f13, State: Initialized, Role: FOLLOWER
I20260927 06:59:32.531600 9935 consensus_queue.cc:260] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } }
I20260927 06:59:32.531826 9941 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } } }
I20260927 06:59:32.531872 9935 sys_catalog.cc:564] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:32.531919 9941 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 06:59:32.533339 9952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:32.533397 9952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:32.534775 9867 master_runner.cc:428] Detected that this master a555d7cefc394026a632b51a9d409f13 is joining an existing cluster
I20260927 06:59:32.534813 9867 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.57.61:42163
I20260927 06:59:32.541970 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.536241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38684)
0927 06:59:32.536347 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.536356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.536406 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.536540 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.536541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.536709 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.536770 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.536771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.537509 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.537510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.538027 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.538028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.538440 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.538512 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.538871 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.538876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.539238 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.540972 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.540973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.540973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.541128 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.541311 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.541312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.541369 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.541650 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.541652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.541694 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.541769 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.541860 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":66,"threads_started":1}
I20260927 06:59:32.542217 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.536266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.3.57.61:42163)
0927 06:59:32.536299 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.536301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.536410 (+ 109us) server_negotiation.cc:407] Connection header received
0927 06:59:32.536461 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.536461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.536469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.536512 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.536855 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.537453 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538102 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.538349 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538407 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.539310 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.539311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.539311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.539316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.540886 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541201 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.541202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.541203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.541247 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541447 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.541448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.541448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.541580 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.541643 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.541789 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.541919 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":17}
I20260927 06:59:32.542433 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.535713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37106)
0927 06:59:32.535886 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.535897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.535968 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.536114 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.536115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.536276 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.536399 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.536401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.537320 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.537321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.538194 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.538197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.538638 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538849 (+ 211us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.538899 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.539251 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.539255 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.539700 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.541443 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.541444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.541642 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.541812 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.541813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.541862 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.542123 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.542124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.542173 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.542243 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.542348 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260927 06:59:32.542447 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.535768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.3.57.62:41287)
0927 06:59:32.535820 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.535822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.535963 (+ 141us) server_negotiation.cc:407] Connection header received
0927 06:59:32.536022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.536023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.536033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.536103 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.536503 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.537241 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538272 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.538544 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.538628 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.539788 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.539789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.539790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.539796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.541361 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541695 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.541696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.541697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.541745 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.541932 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.541932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.541933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.542043 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.542081 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.542287 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.542362 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.62@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
I20260927 06:59:32.551026 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.544302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.3.57.61:42163)
0927 06:59:32.544339 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.544340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.544470 (+ 130us) server_negotiation.cc:407] Connection header received
0927 06:59:32.544524 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.544525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.544533 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.544597 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.544898 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.545566 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.546266 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.546556 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.546623 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.548759 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.548760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.548761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.548766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.550018 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.550433 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.550434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.550434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.550475 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.550628 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.550629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.550629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.550706 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.550751 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.550929 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.550979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:32.551167 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.544261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38686)
0927 06:59:32.544377 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.544411 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.544474 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.544599 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.544601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.544801 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.544840 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.544841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.545636 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.545637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.546186 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.546188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.546623 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.546624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.546673 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.547124 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.547129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.548660 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.550089 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.550096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.550098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.550367 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.550515 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.550516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.550516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.550575 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.550765 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.550771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.550821 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.550900 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.551014 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
I20260927 06:59:32.551663 9401 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.57.61:42163: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a555d7cefc394026a632b51a9d409f13"
member_type: NON_VOTER
last_known_addr {
host: "127.3.57.61"
port: 42163
}
attrs {
promote: true
}
}
dest_uuid: "57f2e6c4243c434d9b1d25ff9653e653"
cas_config_opid_index: -1
I20260927 06:59:32.551959 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } }
I20260927 06:59:32.552377 9962 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a555d7cefc394026a632b51a9d409f13 (127.3.57.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } }
I20260927 06:59:32.552634 9909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:32.553313 9962 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.57.61:42163
I20260927 06:59:32.553316 9909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Refusing update from remote peer 57f2e6c4243c434d9b1d25ff9653e653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 06:59:32.553450 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } }
I20260927 06:59:32.553517 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.553529 9867 master.cc:537] Master@127.3.57.61:42163 shutting down...
I20260927 06:59:32.553757 9962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 06:59:32.554387 9389 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 06:59:32.555290 9389 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 -> Peer a555d7cefc394026a632b51a9d409f13 (127.3.57.61:42163): Couldn't send request to peer a555d7cefc394026a632b51a9d409f13. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 06:59:32.556025 9867 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:32.556103 9867 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:32.556155 9867 tablet_replica.cc:330] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: stopping tablet replica
I20260927 06:59:32.568444 9867 master.cc:552] Master@127.3.57.61:42163 shutdown complete.
I20260927 06:59:32.568490 9867 master_runner.cc:305] Clearing existing system tablet
I20260927 06:59:32.568614 9867 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.568660 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.569310 9867 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:32.569804 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.569943 9867 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.570003 9867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.570029 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "a555d7cefc394026a632b51a9d409f13"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b16721bc1fb6ab8a4f38d960ef0c0e30"
server_key_iv: "4fb730a562a35da7d34fa56e38a8537b"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.570778 9867 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:32.572275 9867 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 06:59:32.572342 9867 log.cc:1196] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 06:59:32.572537 9867 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Deleting consensus metadata
I20260927 06:59:32.572871 9867 master_runner.cc:315] Copying system tablet from 127.3.57.62:41287
I20260927 06:59:32.573427 9867 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: tablet copy: Beginning tablet copy session from remote peer at address 127.3.57.62:41287
I20260927 06:59:32.579203 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.573672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37120)
0927 06:59:32.573801 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.573809 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.573857 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.573984 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.573985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.574107 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.574158 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.574159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.575098 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.575099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.575691 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.575693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.576138 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.576138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.576180 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.576574 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.576577 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.576819 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.578397 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.578397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.578398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.578557 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.578734 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.578735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.578735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.578779 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.579005 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.579006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.579036 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.579081 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.579143 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260927 06:59:32.579250 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.573720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.3.57.62:41287)
0927 06:59:32.573759 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.573761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.573863 (+ 102us) server_negotiation.cc:407] Connection header received
0927 06:59:32.573908 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.573908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.573918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.573983 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.574221 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.575028 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.575762 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.576070 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.576141 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.576896 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.576898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.576898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.576905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.578325 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.578620 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.578621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.578621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.578676 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.578826 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.578826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.578827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.578947 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.579003 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.579101 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.579167 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 06:59:32.579484 9423 tablet_copy_service.cc:140] P 57f2e6c4243c434d9b1d25ff9653e653: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a555d7cefc394026a632b51a9d409f13 ({username='oryx', principal='oryx/127.3.57.61@KRBTEST.COM'} at 127.0.0.1:37120)
I20260927 06:59:32.579586 9423 tablet_copy_service.cc:161] P 57f2e6c4243c434d9b1d25ff9653e653: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a555d7cefc394026a632b51a9d409f13 at {username='oryx', principal='oryx/127.3.57.61@KRBTEST.COM'} at 127.0.0.1:37120: session id = a555d7cefc394026a632b51a9d409f13-00000000000000000000000000000000
I20260927 06:59:32.580183 9423 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 06:59:32.580703 9867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 06:59:32.581729 9867 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: tablet copy: Starting download of 0 data blocks...
I20260927 06:59:32.581867 9867 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: tablet copy: Starting download of 1 WAL segments...
I20260927 06:59:32.582612 9867 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 06:59:32.583460 9423 tablet_copy_service.cc:342] P 57f2e6c4243c434d9b1d25ff9653e653: Request end of tablet copy session a555d7cefc394026a632b51a9d409f13-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.57.61@KRBTEST.COM'} at 127.0.0.1:37120
I20260927 06:59:32.583518 9423 tablet_copy_service.cc:433] P 57f2e6c4243c434d9b1d25ff9653e653: ending tablet copy session a555d7cefc394026a632b51a9d409f13-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a555d7cefc394026a632b51a9d409f13
I20260927 06:59:32.585747 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.586776 9973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.586815 9976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.586896 9974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.587706 9867 server_base.cc:1034] running on GCE node
I20260927 06:59:32.587803 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.587922 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.589066 9867 hybrid_clock.cc:648] HybridClock initialized: now 1790492372589053 us; error 24 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.590210 9867 init.cc:377] Logged in from keytab as oryx/127.3.57.61@KRBTEST.COM (short username oryx)
I20260927 06:59:32.590499 9867 webserver.cc:460] Webserver started at http://127.3.57.61:44341/ using document root <none> and password file <none>
I20260927 06:59:32.590612 9867 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.590660 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.591208 9867 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.591701 9983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.591812 9867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.591859 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "a555d7cefc394026a632b51a9d409f13"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b16721bc1fb6ab8a4f38d960ef0c0e30"
server_key_iv: "4fb730a562a35da7d34fa56e38a8537b"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.591907 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.600313 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.601861 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.604856 9867 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.61:42163
I20260927 06:59:32.605067 10035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.61:42163 every 8 connection(s)
I20260927 06:59:32.605211 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
I20260927 06:59:32.605989 10036 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:59:32.606472 10036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Bootstrap starting.
I20260927 06:59:32.609241 10036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:59:32.609494 10036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: Bootstrap complete.
I20260927 06:59:32.609685 10036 raft_consensus.cc:348] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } }
I20260927 06:59:32.609767 10036 raft_consensus.cc:729] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a555d7cefc394026a632b51a9d409f13, State: Initialized, Role: LEARNER
I20260927 06:59:32.609803 10036 consensus_queue.cc:260] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } }
I20260927 06:59:32.609963 10036 sys_catalog.cc:564] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:32.609961 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } }
I20260927 06:59:32.610070 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:59:32.611538 10050 catalog_manager.cc:1259] Loaded cluster ID: 48dab597f0b248e0b7a32b05db3b36d3
I20260927 06:59:32.611610 10050 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: loading cluster ID for follower catalog manager: success
I20260927 06:59:32.612666 10050 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: acquiring CA information for follower catalog manager: success
I20260927 06:59:32.612960 10050 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 06:59:32.629647 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.623034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.3.57.61:42163)
0927 06:59:32.623161 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.623163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.623173 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:32.623196 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.623197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.623207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.623294 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.623846 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.624852 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.625568 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.625881 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.625979 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.627029 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.627030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.627030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.627037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.628612 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.628916 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.628917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.628918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.628988 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.629163 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.629164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.629165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.629292 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.629395 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.629471 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.629547 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
WARNING: no policy specified for oryx/127.3.57.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.60.keytab.
Entry for principal oryx/127.3.57.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.60.keytab.
I20260927 06:59:32.651976 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:46671
--webserver_interface=127.3.57.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.60.keytab
--principal=oryx/127.3.57.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:32.732990 10054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.733173 10054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.733193 10054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.734391 10054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 06:59:32.734432 10054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.734447 10054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.734459 10054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 06:59:32.734472 10054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 06:59:32.736202 10054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.60.keytab
--principal=oryx/127.3.57.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.57.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:32.736442 10054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.736671 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.741370 10062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.741384 10060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.741406 10059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.741556 10054 server_base.cc:1034] running on GCE node
I20260927 06:59:32.741801 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.742081 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.743261 10054 hybrid_clock.cc:648] HybridClock initialized: now 1790492372743253 us; error 54 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.745656 10054 init.cc:377] Logged in from keytab as oryx/127.3.57.60@KRBTEST.COM (short username oryx)
I20260927 06:59:32.746052 10054 webserver.cc:460] Webserver started at http://127.3.57.60:37923/ using document root <none> and password file <none>
I20260927 06:59:32.746340 10054 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.746409 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.746560 10054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 06:59:32.747879 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/instance:
uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "a25f2984673e345af1786a62776b36c5"
server_key_iv: "546ff28384e0c0e0458b2cc11f000e4f"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.748240 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal/instance:
uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "a25f2984673e345af1786a62776b36c5"
server_key_iv: "546ff28384e0c0e0458b2cc11f000e4f"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.749521 10054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 06:59:32.750267 10069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.750460 10054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:32.750557 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "a25f2984673e345af1786a62776b36c5"
server_key_iv: "546ff28384e0c0e0458b2cc11f000e4f"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.750646 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.759644 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.761296 10054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.761442 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.765244 10054 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.60:46671
I20260927 06:59:32.765275 10121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.60:46671 every 8 connection(s)
I20260927 06:59:32.765702 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
I20260927 06:59:32.766031 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10054
I20260927 06:59:32.766808 10122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:32.768736 10122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.60@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:32.777030 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.766652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.3.57.60:46671)
0927 06:59:32.766844 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.766852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.766883 (+ 31us) server_negotiation.cc:407] Connection header received
0927 06:59:32.767283 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.767289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.767352 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.767456 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.768253 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.769163 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.770184 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.770818 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.770925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.773680 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.773693 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.773693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.773698 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.775359 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.775767 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.775770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.775771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.775872 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.776063 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.776064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.776064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.776368 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.776488 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.776624 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.776758 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260927 06:59:32.777519 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.769427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.3.57.62:41287)
0927 06:59:32.769472 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.769474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.769534 (+ 60us) server_negotiation.cc:407] Connection header received
0927 06:59:32.769608 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.769609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.769616 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.769657 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.770244 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.770993 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.772299 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.772484 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.772559 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.774618 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.774620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.774620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.774628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.776287 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.776709 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.776711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.776711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.776769 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.776959 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.776960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.776960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.777090 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.777160 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.777362 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.777430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 06:59:32.777534 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.769286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37130)
0927 06:59:32.769450 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.769487 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.769551 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.769681 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.769683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.770082 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.770162 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.770164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.771078 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.771079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.772202 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.772205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.772569 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.772570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.772637 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.773082 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.773089 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.774548 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.776366 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.776371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.776372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.776635 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.776835 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.776835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.776836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.776899 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.777187 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.777192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.777237 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.777320 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.777424 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260927 06:59:32.777974 10122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.60@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
I20260927 06:59:32.783793 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.778203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38712)
0927 06:59:32.778249 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.778259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.778294 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.778426 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.778426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.778561 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.778598 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.778600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.779223 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.779224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.779781 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.779782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.780158 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.780159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.780221 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.780530 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.780533 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.781548 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.782810 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.782811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.782812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.782987 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.783191 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.783192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.783192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.783249 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.783491 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.783491 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.783528 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.783618 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.783712 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 06:59:32.783824 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.778277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.3.57.61:42163)
0927 06:59:32.778321 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.778323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.778332 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:32.778357 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.778358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.778369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.778424 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.778673 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.779133 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.779870 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.780079 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.780158 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.781631 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.781632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.781633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.781639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.782723 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.783069 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.783070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.783071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.783127 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.783313 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.783314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.783314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.783417 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.783499 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.783653 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.783722 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:32.784312 10122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 46671 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.60@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
I20260927 06:59:32.789834 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.784514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.0.0.1:54228)
0927 06:59:32.784559 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.784570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.784607 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.784730 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.784731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.784827 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.784850 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.784851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.785440 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.785440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.785872 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.785873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.786201 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.786202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.786235 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.786533 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.786676 (+ 143us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.787796 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.789032 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.789033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.789033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.789181 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.789384 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.789385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.789385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.789430 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.789620 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.789621 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.789649 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.789696 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.789784 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 06:59:32.789971 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.784602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.3.57.60:46671)
0927 06:59:32.784648 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.784650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.784665 (+ 15us) server_negotiation.cc:407] Connection header received
0927 06:59:32.784681 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.784681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.784687 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.784729 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.784926 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.785372 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.785953 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.786136 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.786193 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.787877 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.787878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.787878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.787883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.788969 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.789262 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.789263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.789263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.789311 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.789486 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.789487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.789487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.789569 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.789615 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.789712 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.789849 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":30}
I20260927 06:59:32.791153 10122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Bootstrap starting.
I20260927 06:59:32.792030 10122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:32.792300 10122 log.cc:824] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:32.792979 10122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: No bootstrap required, opened a new log
I20260927 06:59:32.794343 10122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 46671 } }
I20260927 06:59:32.794450 10122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:32.794471 10122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddcf2a7cfb540e2a1a76c2d57744ab2, State: Initialized, Role: FOLLOWER
I20260927 06:59:32.794603 10122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 46671 } }
I20260927 06:59:32.794834 10127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 46671 } } }
I20260927 06:59:32.794884 10122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:32.794899 10127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 06:59:32.796648 10138 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 06:59:32.796705 10138 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 06:59:32.798955 10054 master_runner.cc:428] Detected that this master 4ddcf2a7cfb540e2a1a76c2d57744ab2 is joining an existing cluster
I20260927 06:59:32.798998 10054 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.57.60:46671
I20260927 06:59:32.807132 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.800197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.3.57.61:42163)
0927 06:59:32.800239 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.800241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.800455 (+ 214us) server_negotiation.cc:407] Connection header received
0927 06:59:32.800557 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.800558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.800566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.800593 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.801267 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.801972 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803133 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803409 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803449 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.804534 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.804536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.804536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.804542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.806080 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.806488 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.806489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.806489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.806538 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.806722 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.806723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.806723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.806814 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.806862 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.807013 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.807069 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:32.807189 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.800038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38724)
0927 06:59:32.800393 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.800403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.800478 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.800652 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.800652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.801104 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.801177 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.801179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.802065 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.802066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803036 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.803038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803532 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803643 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.804030 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.804035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.804441 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.806161 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.806162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.806162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.806386 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.806600 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.806601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.806601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.806664 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.806874 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.806874 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.806918 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.806996 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.807089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":102,"thread_start_us":77,"threads_started":1}
I20260927 06:59:32.828393 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9465
I20260927 06:59:32.831976 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.800189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.3.57.62:41287)
0927 06:59:32.800231 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.800233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.800340 (+ 107us) server_negotiation.cc:407] Connection header received
0927 06:59:32.800451 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.800452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.800464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.800500 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.801047 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.801893 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803203 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803530 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.828876 (+ 25252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.828878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.828878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.828885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.830711 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.831093 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.831094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.831095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.831145 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.831331 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.831332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.831332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.831482 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.831526 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.831754 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.831844 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:32.832350 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.799998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37132)
0927 06:59:32.800273 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.800283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.800366 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.800577 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.800578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.800848 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.800927 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.800929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.801985 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.801985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803115 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.803117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803696 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803751 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.828310 (+ 24559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.828314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.828762 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.830797 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.830798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.830799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.831013 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.831211 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.831211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.831212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.831269 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.831591 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.831592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.831640 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.832064 (+ 424us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.832188 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":92,"thread_start_us":75,"threads_started":1}
I20260927 06:59:32.833954 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.800055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.3.57.60:46671)
0927 06:59:32.800092 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.800094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.800456 (+ 362us) server_negotiation.cc:407] Connection header received
0927 06:59:32.800557 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.800557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.800566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.800593 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.801260 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.801994 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803132 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803438 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803476 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.830124 (+ 26648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.830126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.830126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.830132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.832790 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.833195 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.833196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.833197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.833243 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.833422 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.833423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.833423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.833551 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.833589 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.833751 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.833828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 06:59:32.834290 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.800133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.0.0.1:54240)
0927 06:59:32.800401 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.800412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.800476 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.800646 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.800646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.801121 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.801178 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.801180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.802080 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.802081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803039 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.803041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.803570 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.803571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.803655 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.828231 (+ 24576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.828242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.830025 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.832887 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.832888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.832889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.833111 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.833301 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.833302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.833302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.833365 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.833624 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.833625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.833659 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.834054 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.834146 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1,"thread_start_us":203,"threads_started":1}
I20260927 06:59:32.835856 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:39683
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:41287
--webserver_port=41675
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492371, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.62@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
I20260927 06:59:32.842724 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.835119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.3.57.60:46671)
0927 06:59:32.835167 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.835169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.835199 (+ 30us) server_negotiation.cc:407] Connection header received
0927 06:59:32.835265 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.835266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.835276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.835330 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.835631 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.836322 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.837322 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.837592 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.837663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.839799 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.839800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.839801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.839807 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.841480 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.841903 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.841904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.841905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.841963 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.842175 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.842176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.842176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.842306 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.842369 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.842576 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.842644 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 06:59:32.842772 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.835055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.0.0.1:54250)
0927 06:59:32.835158 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.835168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.835212 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.835348 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.835349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.835502 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.835565 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.835567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.836407 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.836408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.837236 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.837239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.837669 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.837671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.837739 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.838225 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.838230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.839709 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.841568 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.841569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.841569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.841808 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.842034 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.842035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.842035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.842098 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.842386 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.842387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.842446 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.842530 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.842639 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 06:59:32.843603 9401 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.57.60:46671: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
member_type: NON_VOTER
last_known_addr {
host: "127.3.57.60"
port: 46671
}
attrs {
promote: true
}
}
dest_uuid: "57f2e6c4243c434d9b1d25ff9653e653"
cas_config_opid_index: 5
I20260927 06:59:32.843847 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:32.844388 9962 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ddcf2a7cfb540e2a1a76c2d57744ab2 (127.3.57.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.845419 9962 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.57.60:46671
I20260927 06:59:32.845500 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.845589 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.845909 10054 master.cc:537] Master@127.3.57.60:46671 shutting down...
I20260927 06:59:32.846046 10096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:32.847723 10096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 1 FOLLOWER]: Refusing update from remote peer 57f2e6c4243c434d9b1d25ff9653e653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 06:59:32.848048 9961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 06:59:32.848778 9390 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 -> Peer 4ddcf2a7cfb540e2a1a76c2d57744ab2 (127.3.57.60:46671): Couldn't send request to peer 4ddcf2a7cfb540e2a1a76c2d57744ab2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 06:59:32.849258 10054 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:32.849350 10054 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:32.849390 10054 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: stopping tablet replica
I20260927 06:59:32.852360 10054 master.cc:552] Master@127.3.57.60:46671 shutdown complete.
I20260927 06:59:32.852399 10054 master_runner.cc:305] Clearing existing system tablet
I20260927 06:59:32.852519 10054 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.852555 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.853109 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.844406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.3.57.61:42163)
0927 06:59:32.845666 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.845668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.848833 (+ 3165us) server_negotiation.cc:407] Connection header received
0927 06:59:32.848938 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.848939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.848960 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.849000 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:32.849289 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.850084 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.852069 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.852661 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.852704 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.852758 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.853008 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.853045 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234}
I20260927 06:59:32.853312 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.844341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.0.0.1:38734)
0927 06:59:32.848784 (+ 4443us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.848797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.848863 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.849074 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.849074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.849150 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:32.849190 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.849192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.850602 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.850603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.851973 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.851976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.852853 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.852853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.852916 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.852934 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.853215 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.853253 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4426}
I20260927 06:59:32.853369 10054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:32.853525 10010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 LEARNER]: Refusing update from remote peer 57f2e6c4243c434d9b1d25ff9653e653: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 06:59:32.853683 9961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 06:59:32.853955 10149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.854070 10054 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.854111 10054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.854128 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "a25f2984673e345af1786a62776b36c5"
server_key_iv: "546ff28384e0c0e0458b2cc11f000e4f"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.854122 10010 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4ddcf2a7cfb540e2a1a76c2d57744ab2 (127.3.57.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.854790 10054 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:32.855289 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f2e6c4243c434d9b1d25ff9653e653. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.855405 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:59:32.855449 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.855520 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:59:32.856119 10054 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 06:59:32.856184 10054 log.cc:1196] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 06:59:32.856352 10054 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Deleting consensus metadata
I20260927 06:59:32.856676 10054 master_runner.cc:315] Copying system tablet from 127.3.57.62:41287
I20260927 06:59:32.857245 10054 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: tablet copy: Beginning tablet copy session from remote peer at address 127.3.57.62:41287
I20260927 06:59:32.863878 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.857519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37144)
0927 06:59:32.857619 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.857628 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.857698 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.857830 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.857831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.858017 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.858068 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.858070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.858950 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.858951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.859817 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.859819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.860267 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.860268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.860332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.860774 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.860779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.861145 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.862802 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.862803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.862804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.863012 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.863218 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.863219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.863219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.863279 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.863579 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.863581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.863620 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.863691 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.863780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260927 06:59:32.864270 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.857522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.3.57.62:41287)
0927 06:59:32.857566 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.857567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.857705 (+ 138us) server_negotiation.cc:407] Connection header received
0927 06:59:32.857750 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.857751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.857761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.857821 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.858137 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.858868 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.859903 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.860191 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.860257 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.861226 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.861227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.861227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.861233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.862719 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.863088 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.863089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.863089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.863145 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.863344 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.863345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.863345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.863499 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.863576 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.863731 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.863806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 06:59:32.864295 9961 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: attempting to promote NON_VOTER a555d7cefc394026a632b51a9d409f13 to VOTER
I20260927 06:59:32.864564 9423 tablet_copy_service.cc:140] P 57f2e6c4243c434d9b1d25ff9653e653: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ddcf2a7cfb540e2a1a76c2d57744ab2 ({username='oryx', principal='oryx/127.3.57.60@KRBTEST.COM'} at 127.0.0.1:37144)
I20260927 06:59:32.864604 9423 tablet_copy_service.cc:161] P 57f2e6c4243c434d9b1d25ff9653e653: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ddcf2a7cfb540e2a1a76c2d57744ab2 at {username='oryx', principal='oryx/127.3.57.60@KRBTEST.COM'} at 127.0.0.1:37144: session id = 4ddcf2a7cfb540e2a1a76c2d57744ab2-00000000000000000000000000000000
I20260927 06:59:32.864653 9961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:32.865077 9423 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 06:59:32.865208 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.865050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.0.0.1:54254)
0927 06:59:32.865154 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.865185 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:46671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 06:59:32.865453 10010 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 LEARNER]: Refusing update from remote peer 57f2e6c4243c434d9b1d25ff9653e653: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 06:59:32.865608 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 06:59:32.865711 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.865634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.0.0.1:54256)
0927 06:59:32.865668 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.865689 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:46671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 06:59:32.865763 9390 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 -> Peer 4ddcf2a7cfb540e2a1a76c2d57744ab2 (127.3.57.60:46671): Couldn't send request to peer 4ddcf2a7cfb540e2a1a76c2d57744ab2. Status: Network error: Client connection negotiation failed: client connection to 127.3.57.60:46671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 06:59:32.865899 9962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 06:59:32.866539 9962 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a555d7cefc394026a632b51a9d409f13 (127.3.57.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.866641 10054 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: tablet copy: Starting download of 0 data blocks...
I20260927 06:59:32.866760 10054 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: tablet copy: Starting download of 1 WAL segments...
I20260927 06:59:32.867303 9962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.867370 9962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.867384 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.867482 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653 [sys.catalog]: This master's current role is: LEADER
I20260927 06:59:32.867726 10010 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a555d7cefc394026a632b51a9d409f13 (127.3.57.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.868592 10151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57f2e6c4243c434d9b1d25ff9653e653" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.868672 10151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 06:59:32.868893 10054 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 06:59:32.869675 9423 tablet_copy_service.cc:342] P 57f2e6c4243c434d9b1d25ff9653e653: Request end of tablet copy session 4ddcf2a7cfb540e2a1a76c2d57744ab2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.57.60@KRBTEST.COM'} at 127.0.0.1:37144
I20260927 06:59:32.869752 9423 tablet_copy_service.cc:433] P 57f2e6c4243c434d9b1d25ff9653e653: ending tablet copy session 4ddcf2a7cfb540e2a1a76c2d57744ab2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ddcf2a7cfb540e2a1a76c2d57744ab2
I20260927 06:59:32.872026 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.873471 10159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.876803 10160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.877068 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.879714 10054 server_base.cc:1034] running on GCE node
I20260927 06:59:32.879814 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.879938 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.881068 10054 hybrid_clock.cc:648] HybridClock initialized: now 1790492372881047 us; error 34 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.882552 10054 init.cc:377] Logged in from keytab as oryx/127.3.57.60@KRBTEST.COM (short username oryx)
I20260927 06:59:32.882797 10054 webserver.cc:460] Webserver started at http://127.3.57.60:41689/ using document root <none> and password file <none>
I20260927 06:59:32.882923 10054 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.882962 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.883662 10054 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 06:59:32.884179 10169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.884279 10054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.884310 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "a25f2984673e345af1786a62776b36c5"
server_key_iv: "546ff28384e0c0e0458b2cc11f000e4f"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.884358 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.893039 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.894768 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.897639 10054 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.60:46671
I20260927 06:59:32.897986 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
I20260927 06:59:32.900357 10222 sys_catalog.cc:263] Verifying existing consensus state
I20260927 06:59:32.900867 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Bootstrap starting.
I20260927 06:59:32.903708 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 06:59:32.903956 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: Bootstrap complete.
I20260927 06:59:32.904143 10222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:32.904235 10222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ddcf2a7cfb540e2a1a76c2d57744ab2, State: Initialized, Role: LEARNER
I20260927 06:59:32.904276 10222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:32.904448 10222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 06:59:32.904453 10225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: NON_VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: true } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } } }
I20260927 06:59:32.904657 10225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2 [sys.catalog]: This master's current role is: LEARNER
I20260927 06:59:32.905901 10236 catalog_manager.cc:1259] Loaded cluster ID: 48dab597f0b248e0b7a32b05db3b36d3
I20260927 06:59:32.905974 10221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.60:46671 every 8 connection(s)
I20260927 06:59:32.906034 10236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: loading cluster ID for follower catalog manager: success
I20260927 06:59:32.907886 10236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: acquiring CA information for follower catalog manager: success
I20260927 06:59:32.908150 10236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ddcf2a7cfb540e2a1a76c2d57744ab2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 06:59:32.937160 10147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:32.937333 10147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:32.937355 10147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:32.938582 10147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:32.938623 10147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:32.938649 10147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 06:59:32.940117 10147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:39683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=41675
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:32.940325 10147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:32.940536 10147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:32.945549 10244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:32.945576 10241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.945679 10147 server_base.cc:1034] running on GCE node
W20260927 06:59:32.945554 10242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:32.945981 10147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:32.946244 10147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:32.947398 10147 hybrid_clock.cc:648] HybridClock initialized: now 1790492372947387 us; error 28 us; skew 500 ppm
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:32.949945 10147 init.cc:377] Logged in from keytab as oryx/127.3.57.1@KRBTEST.COM (short username oryx)
I20260927 06:59:32.950423 10147 webserver.cc:460] Webserver started at http://127.3.57.1:41675/ using document root <none> and password file <none>
I20260927 06:59:32.950661 10147 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:32.950730 10147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:32.952314 10147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 06:59:32.952780 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.952948 10147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.953037 10147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "da37e7478e5e432a96ee5564f010d2fd"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "2369c194ea69f215f04ce6dd368abbc9"
server_key_iv: "a37cb40a3e727549f1705968350c2940"
server_key_version: "encryptionkey@0"
I20260927 06:59:32.953351 10147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:32.970116 10147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:32.972168 10147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:32.972318 10147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:32.972515 10147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:32.972775 10147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:32.972807 10147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.972828 10147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:32.972867 10147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:32.978170 10147 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:39683
I20260927 06:59:32.978261 10364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:39683 every 8 connection(s)
I20260927 06:59:32.978538 10147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 06:59:32.984442 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10147
I20260927 06:59:32.984748 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9599
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
I20260927 06:59:32.989286 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:52481 (local address 127.3.57.60:46671)
0927 06:59:32.979339 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.979341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.979701 (+ 360us) server_negotiation.cc:407] Connection header received
0927 06:59:32.980607 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.980608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.980620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.980664 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.981318 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.982292 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983219 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.983482 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983540 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.986198 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.986199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.986200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.986206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.987743 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988415 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.988416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.988416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.988468 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988646 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.988648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.988648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.988768 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.988840 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.989108 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.989190 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260927 06:59:32.989349 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:58509 (local address 127.3.57.61:42163)
0927 06:59:32.979671 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.979673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.979797 (+ 124us) server_negotiation.cc:407] Connection header received
0927 06:59:32.980607 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.980609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.980621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.980656 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.981318 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.982041 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983117 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.983397 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983480 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.985996 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.985998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.985998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.986005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.987727 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988271 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.988272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.988272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.988333 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988588 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.988589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.988589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.988741 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.988804 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.989125 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.989209 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 06:59:32.989758 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.3.57.1:52481)
0927 06:59:32.979458 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.979466 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.980492 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.980732 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.980736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.981079 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.981220 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.981226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.982393 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.982393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.983135 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.983136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.983604 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.983683 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.984095 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.984099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.986114 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.987856 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.987857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.987858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.988196 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.988533 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.988534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.988593 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.988867 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.988867 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.988980 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.989087 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.989235 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":30,"thread_start_us":93,"threads_started":1}
I20260927 06:59:32.989769 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.3.57.1:58509)
0927 06:59:32.979746 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.979754 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.980505 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.980735 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.980738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.981066 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.981220 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.981226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.982147 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.982148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.983008 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.983010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.983486 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.983574 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.983967 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.983973 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.985909 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.987806 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.987813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.987814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.988196 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.988414 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.988414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.988415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.988531 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.988832 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.988838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.989001 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.989428 (+ 427us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.989539 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 06:59:32.990437 10367 heartbeater.cc:347] Connected to a master server at 127.3.57.61:42163
I20260927 06:59:32.990541 10367 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.990711 10367 heartbeater.cc:511] Master 127.3.57.61:42163 requested a full tablet report, sending...
I20260927 06:59:32.991261 10000 ts_manager.cc:194] Registered new tserver with Master: da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683)
I20260927 06:59:32.992148 10365 heartbeater.cc:347] Connected to a master server at 127.3.57.60:46671
I20260927 06:59:32.992198 10365 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.992269 10365 heartbeater.cc:511] Master 127.3.57.60:46671 requested a full tablet report, sending...
I20260927 06:59:32.992345 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:42509
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:41287
--webserver_port=35735
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 06:59:32.992620 10186 ts_manager.cc:194] Registered new tserver with Master: da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683)
Sep 27 06:59:32 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:32.996150 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:57967 (local address 127.3.57.62:41287)
0927 06:59:32.979341 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:32.979343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:32.979528 (+ 185us) server_negotiation.cc:407] Connection header received
0927 06:59:32.980591 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:32.980592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:32.980604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:32.980664 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:32.981305 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.981773 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.982747 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.982932 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983028 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.993322 (+ 10294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:32.993323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:32.993324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:32.993339 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:32.994998 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.995324 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.995326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.995326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.995387 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:32.995631 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:32.995632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:32.995632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:32.995751 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:32.995826 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:32.995983 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 06:59:32.996054 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 06:59:32.996326 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:32.979241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:57967)
0927 06:59:32.979409 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:32.979448 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:32.980493 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:32.980733 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:32.980736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:32.981038 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:32.981218 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.981222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.981868 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.981869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.982649 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:32.982650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:32.983033 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:32.983034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:32.983177 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:32.983766 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:32.983775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:32.993225 (+ 9450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:32.995085 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.995086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.995087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.995253 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.995469 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:32.995469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:32.995470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:32.995529 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:32.995818 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:32.995819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:32.995869 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 06:59:32.995947 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 06:59:32.996048 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260927 06:59:32.996799 10366 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:32.996855 10366 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:32.996920 10366 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
I20260927 06:59:32.997174 9401 ts_manager.cc:194] Re-registered known tserver with Master: da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683)
I20260927 06:59:32.997715 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:57967
W20260927 06:59:33.073231 10373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:33.073406 10373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:33.073429 10373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:33.074457 10373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:33.074503 10373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:33.074527 10373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 06:59:33.075927 10373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=35735
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:33.076099 10373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:33.076259 10373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:33.081115 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:33.081135 10378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:33.081152 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:33.081182 10373 server_base.cc:1034] running on GCE node
I20260927 06:59:33.081449 10373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:33.081627 10373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:33.082775 10373 hybrid_clock.cc:648] HybridClock initialized: now 1790492373082769 us; error 50 us; skew 500 ppm
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:33.085242 10373 init.cc:377] Logged in from keytab as oryx/127.3.57.2@KRBTEST.COM (short username oryx)
I20260927 06:59:33.085721 10373 webserver.cc:460] Webserver started at http://127.3.57.2:35735/ using document root <none> and password file <none>
I20260927 06:59:33.085973 10373 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:33.086030 10373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:33.087299 10373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:33.087911 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.088066 10373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.088176 10373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "45d9609bdbcc443abbea3255ff647e8c"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "f235aa41c679bfacf7dbc60987befb7b"
server_key_iv: "780ffcacfd1947d949c36ad24e843eb9"
server_key_version: "encryptionkey@0"
I20260927 06:59:33.088479 10373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:33.101168 10373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:33.102882 10373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:33.103039 10373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:33.103245 10373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:33.103525 10373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:33.103643 10373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.103685 10373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:33.103711 10373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.109956 10373 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:42509
I20260927 06:59:33.110031 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:42509 every 8 connection(s)
I20260927 06:59:33.110391 10373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
I20260927 06:59:33.117291 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10373
I20260927 06:59:33.117432 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9733
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
I20260927 06:59:33.121073 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:48041 (local address 127.3.57.61:42163)
0927 06:59:33.111483 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.111486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.111708 (+ 222us) server_negotiation.cc:407] Connection header received
0927 06:59:33.112255 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.112256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.112268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.112301 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.113141 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.113630 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.114785 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.115016 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115101 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.117464 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.117466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.117467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.117474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.119475 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120008 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.120009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.120010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.120065 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120341 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.120342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.120342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.120472 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.120521 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.120853 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.120933 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260927 06:59:33.121497 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:45639 (local address 127.3.57.60:46671)
0927 06:59:33.111520 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.111522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.111684 (+ 162us) server_negotiation.cc:407] Connection header received
0927 06:59:33.112239 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.112241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.112256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.112300 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.113747 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.114488 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115348 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.115669 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115737 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.118556 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.118557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.118557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.118564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.120284 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120679 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.120680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.120680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.120730 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120935 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.120936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.120936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.121059 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.121097 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.121312 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.121403 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 06:59:33.121589 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:56147 (local address 127.3.57.62:41287)
0927 06:59:33.111446 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.111449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.111699 (+ 250us) server_negotiation.cc:407] Connection header received
0927 06:59:33.112256 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.112258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.112269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.112301 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.113161 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.113665 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115159 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.115475 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115557 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.118539 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.118540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.118541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.118549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.120445 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120793 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.120794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.120794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.120844 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.121032 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.121033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.121033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.121177 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.121217 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.121423 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.121491 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 06:59:33.121658 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.3.57.2:48041)
0927 06:59:33.111652 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.111662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.112137 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.112396 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.112396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.112874 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.113051 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.113056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.113735 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.113736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.114702 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.114704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.115110 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.115236 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.115786 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.115796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.117346 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.119582 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.119588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.119590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.119912 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.120134 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.120135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.120260 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.120552 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.120557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.120708 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.121200 (+ 492us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.121397 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":435,"thread_start_us":81,"threads_started":1}
I20260927 06:59:33.121806 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.3.57.2:45639)
0927 06:59:33.111533 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.111609 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.112150 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.112644 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.112645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.112863 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.113049 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.113053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.114564 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.114565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.115267 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.115269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.116376 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.116377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.116421 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.116818 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.116825 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.118470 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.120376 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.120378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.120601 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.120801 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.120802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.120864 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.121160 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.121160 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.121200 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.121596 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.121694 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":88,"threads_started":1}
I20260927 06:59:33.121896 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.111127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:56147)
0927 06:59:33.111629 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.111640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.112137 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.112382 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.112387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.112859 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.113050 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.113056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.113749 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.113751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.114697 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.114699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.115572 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.115573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.115656 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.116094 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.116098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.118419 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.120520 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.120521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.120715 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.120914 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.120915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.120915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.120965 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.121263 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.121263 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.121298 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.121681 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.121749 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":49,"thread_start_us":77,"threads_started":1}
I20260927 06:59:33.122152 10504 heartbeater.cc:347] Connected to a master server at 127.3.57.61:42163
I20260927 06:59:33.122231 10504 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.122413 10504 heartbeater.cc:511] Master 127.3.57.61:42163 requested a full tablet report, sending...
I20260927 06:59:33.122728 10503 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:33.122763 10503 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.122807 10503 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
I20260927 06:59:33.122843 10000 ts_manager.cc:194] Registered new tserver with Master: 45d9609bdbcc443abbea3255ff647e8c (127.3.57.2:42509)
I20260927 06:59:33.123049 9401 ts_manager.cc:194] Re-registered known tserver with Master: 45d9609bdbcc443abbea3255ff647e8c (127.3.57.2:42509)
I20260927 06:59:33.123409 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:56147
I20260927 06:59:33.124818 10502 heartbeater.cc:347] Connected to a master server at 127.3.57.60:46671
I20260927 06:59:33.124886 10502 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.124960 10502 heartbeater.cc:511] Master 127.3.57.60:46671 requested a full tablet report, sending...
I20260927 06:59:33.125262 10186 ts_manager.cc:194] Registered new tserver with Master: 45d9609bdbcc443abbea3255ff647e8c (127.3.57.2:42509)
I20260927 06:59:33.126361 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:44451
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:41287
--webserver_port=40905
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 06:59:33.205638 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 06:59:33.205870 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 06:59:33.205910 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 06:59:33.207185 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 06:59:33.207258 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 06:59:33.207314 10509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 06:59:33.208813 10509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:44451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=40905
--tserver_master_addrs=127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 06:59:33.209060 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 06:59:33.209290 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 06:59:33.214612 10515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 06:59:33.214664 10514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:33.214738 10509 server_base.cc:1034] running on GCE node
W20260927 06:59:33.214613 10517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 06:59:33.215093 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 06:59:33.215350 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 06:59:33.216495 10509 hybrid_clock.cc:648] HybridClock initialized: now 1790492373216474 us; error 38 us; skew 500 ppm
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 06:59:33.218856 10509 init.cc:377] Logged in from keytab as oryx/127.3.57.3@KRBTEST.COM (short username oryx)
I20260927 06:59:33.219208 10509 webserver.cc:460] Webserver started at http://127.3.57.3:40905/ using document root <none> and password file <none>
I20260927 06:59:33.219380 10509 fs_manager.cc:359] Metadata directory not provided
I20260927 06:59:33.219429 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 06:59:33.220861 10509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 06:59:33.221421 10524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.221617 10509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 06:59:33.221693 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "aa73dc4a1b664debaa59feb5eaa7e65b"
format_stamp: "Formatted at 2026-09-27 06:59:32 on dist-test-slave-mk3h"
server_key: "b752e740d77759eca68ba1f5fa591d1d"
server_key_iv: "ed868115ced14099c8bacec661d65b0e"
server_key_version: "encryptionkey@0"
I20260927 06:59:33.222021 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 06:59:33.240252 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 06:59:33.242200 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 06:59:33.242338 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 06:59:33.242570 10509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 06:59:33.242865 10509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 06:59:33.242923 10509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.242964 10509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 06:59:33.242992 10509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 06:59:33.248939 10509 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:44451
I20260927 06:59:33.248982 10637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:44451 every 8 connection(s)
I20260927 06:59:33.249305 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 06:59:33.251135 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10509
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.60@KRBTEST.COM
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.61@KRBTEST.COM
Sep 27 06:59:33 dist-test-slave-mk3h krb5kdc[9358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 06:59:33.261531 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.252740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.3.57.60:46671)
0927 06:59:33.253121 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.253122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.253232 (+ 110us) server_negotiation.cc:407] Connection header received
0927 06:59:33.253320 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.253321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.253333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.253370 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.254038 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.254738 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.256320 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.256639 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.256693 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.258229 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.258230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.258231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.258237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.260002 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.260440 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.260441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.260441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.260495 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.260685 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.260686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.260686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.260814 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.260923 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.261332 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.261405 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":66,"threads_started":1}
I20260927 06:59:33.262745 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.249963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:42163 (local address 127.3.57.3:48579)
0927 06:59:33.250394 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.250403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.251384 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.251759 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.251760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.252280 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.252517 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.252524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.253484 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.253485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.254731 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.254733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.255253 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.255254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.255386 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.256070 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.256076 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.258740 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.261104 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.261110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.261112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.261410 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.261624 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.261624 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.261625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.261754 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.262063 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.262069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.262227 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.262337 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.262495 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":76,"thread_start_us":85,"threads_started":1}
I20260927 06:59:33.263159 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.250156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:48579 (local address 127.3.57.61:42163)
0927 06:59:33.250200 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.250202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.250450 (+ 248us) server_negotiation.cc:407] Connection header received
0927 06:59:33.251503 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.251505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.251530 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.251580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.252668 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.253367 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.254824 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.255134 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.255190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.258866 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.258867 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.258867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.258874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.260718 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.261471 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.261473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.261473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.261540 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.261829 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.261830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.261830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.261974 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.262034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.262973 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.263045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 06:59:33.263500 10640 heartbeater.cc:347] Connected to a master server at 127.3.57.61:42163
I20260927 06:59:33.263614 10640 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.263741 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.250106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:57537 (local address 127.3.57.60:46671)
0927 06:59:33.250276 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.250278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.250288 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:33.251530 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.251531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.251544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.251591 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.252623 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.253362 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.254824 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.255134 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.255190 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.259785 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.259786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.259787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.259793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.262121 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.262467 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.262468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.262468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.262521 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.262720 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.262721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.262721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.262851 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.262906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.263581 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.263657 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 06:59:33.263789 10640 heartbeater.cc:511] Master 127.3.57.61:42163 requested a full tablet report, sending...
I20260927 06:59:33.263975 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.249963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:40003)
0927 06:59:33.250490 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.250497 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.251383 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.251759 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.251760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.252274 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.255627 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.255629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.257127 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.257128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.257936 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.257938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.258338 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.258339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.258407 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.258869 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.258874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.260479 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.262902 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.262903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.262903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.263095 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.263303 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.263304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.263304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.263366 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.263689 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.263689 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.263729 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.263791 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.263879 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1588,"thread_start_us":109,"threads_started":1}
I20260927 06:59:33.264184 10000 ts_manager.cc:194] Registered new tserver with Master: aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:33.264305 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.250132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:40003 (local address 127.3.57.62:41287)
0927 06:59:33.250206 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.250208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.250558 (+ 350us) server_negotiation.cc:407] Connection header received
0927 06:59:33.251503 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.251505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.251530 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.251580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.255687 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.256435 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.258024 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.258257 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.258323 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.260582 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.260584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.260584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.260590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.262810 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.263173 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.263174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.263175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.263230 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.263432 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.263433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.263433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.263597 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.263662 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.264128 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.264200 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 06:59:33.264564 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.249963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:46671 (local address 127.3.57.3:57537)
0927 06:59:33.250137 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.250175 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.251370 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.251755 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.251760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.252244 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 06:59:33.252517 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.252523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.253484 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.253485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.254730 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.254733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.255251 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.255252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.255386 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.256008 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 06:59:33.256017 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 06:59:33.259388 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 06:59:33.262179 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.262180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.262181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.262393 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.262592 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 06:59:33.262593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 06:59:33.262594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 06:59:33.262655 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 06:59:33.263123 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 06:59:33.263124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 06:59:33.263165 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.263414 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.263498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":44,"thread_start_us":328,"threads_started":1}
I20260927 06:59:33.264583 10639 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41287
I20260927 06:59:33.264628 10639 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.264693 10639 heartbeater.cc:511] Master 127.3.57.62:41287 requested a full tablet report, sending...
I20260927 06:59:33.264899 9401 ts_manager.cc:194] Re-registered known tserver with Master: aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:33.265235 9401 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:40003
I20260927 06:59:33.265522 10649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.252489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.3.57.62:41287)
0927 06:59:33.252762 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.252764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.252931 (+ 167us) server_negotiation.cc:407] Connection header received
0927 06:59:33.252956 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.252957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.252968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.253052 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.253863 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.254638 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.256501 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.256800 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.256842 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.258862 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.258863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.258863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.258870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.262129 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.264778 (+ 2649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.264779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.264780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.264839 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.265016 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.265018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.265018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.265127 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.265237 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.265338 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.265417 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":72,"threads_started":1}
I20260927 06:59:33.265545 10638 heartbeater.cc:347] Connected to a master server at 127.3.57.60:46671
I20260927 06:59:33.265586 10638 heartbeater.cc:464] Registering TS with master...
I20260927 06:59:33.265637 10638 heartbeater.cc:511] Master 127.3.57.60:46671 requested a full tablet report, sending...
I20260927 06:59:33.265870 10186 ts_manager.cc:194] Registered new tserver with Master: aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:33.267170 9401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 06:59:33.269119 9401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 06:59:33.269950 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.252590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.3.57.61:42163)
0927 06:59:33.255816 (+ 3226us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.255818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.262194 (+ 6376us) server_negotiation.cc:407] Connection header received
0927 06:59:33.262216 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.262217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.262228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.262286 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.262846 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.263610 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.266233 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.266555 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.266623 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.267643 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.267644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.267645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.267650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.268887 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.269214 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.269215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.269215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.269267 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.269487 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.269488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.269488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.269596 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.269697 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.269785 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.269870 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3208,"thread_start_us":48,"threads_started":1}
I20260927 06:59:33.276160 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39683 (local address 127.0.0.1:35380)
0927 06:59:33.272745 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.272754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.272788 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.273364 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.273365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.273367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:33.273404 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.273406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.274205 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.274206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.274957 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.274980 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.275913 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.275914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.275980 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.275985 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.276045 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.276086 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260927 06:59:33.276170 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.3.57.1:39683)
0927 06:59:33.273214 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.273218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.273227 (+ 9us) server_negotiation.cc:407] Connection header received
0927 06:59:33.273262 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.273264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.273280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.273366 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:33.273459 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.274112 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.275077 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.275813 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.275900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.275970 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.276057 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.276095 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":83,"threads_started":1}
I20260927 06:59:33.276623 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:42509 (local address 127.0.0.1:58102)
0927 06:59:33.272613 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.272623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.272664 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.273128 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.273129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.273130 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:33.273164 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.273165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.274067 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.274068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.275202 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.275205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.276400 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.276401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.276461 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.276465 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.276522 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.276564 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 06:59:33.276870 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.3.57.2:42509)
0927 06:59:33.272899 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.272904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.272914 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:33.272951 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.272953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.272967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.273253 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:33.273312 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.273984 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.275295 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.275936 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.276009 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.276366 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.276753 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.276794 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260927 06:59:33.277061 10299 tablet_service.cc:1467] Processing CreateTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 (DEFAULT_TABLE table=test-table [id=45f390ebc29d433aa207766cf685bd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:33.277437 10299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f52a6ec0e12487abdced2d1ad827b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:33.277693 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:44451 (local address 127.0.0.1:33492)
0927 06:59:33.273032 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.273040 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.273078 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.274262 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.274263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.274265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:33.274298 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.274299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.275423 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.275424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.276437 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.276439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.277496 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.277497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.277560 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.277564 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.277620 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.277648 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":58,"threads_started":1}
I20260927 06:59:33.277740 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.272815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.3.57.3:44451)
0927 06:59:33.274130 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.274134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.274142 (+ 8us) server_negotiation.cc:407] Connection header received
0927 06:59:33.274171 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.274173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.274187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.274253 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:33.274364 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.275353 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.276520 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.277380 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.277451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.277520 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.277642 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.277677 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":78,"threads_started":1}
I20260927 06:59:33.280423 10664 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Bootstrap starting.
I20260927 06:59:33.280439 10572 tablet_service.cc:1467] Processing CreateTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 (DEFAULT_TABLE table=test-table [id=45f390ebc29d433aa207766cf685bd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:33.280454 10436 tablet_service.cc:1467] Processing CreateTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 (DEFAULT_TABLE table=test-table [id=45f390ebc29d433aa207766cf685bd54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 06:59:33.280797 10436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f52a6ec0e12487abdced2d1ad827b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:33.280807 10572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f52a6ec0e12487abdced2d1ad827b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 06:59:33.281064 10664 tablet_bootstrap.cc:654] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:33.281376 10664 log.cc:824] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:33.282183 10664 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: No bootstrap required, opened a new log
I20260927 06:59:33.282253 10664 ts_tablet_manager.cc:1397] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:33.283892 10666 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Bootstrap starting.
I20260927 06:59:33.284142 10667 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Bootstrap starting.
I20260927 06:59:33.284606 10666 tablet_bootstrap.cc:654] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:33.284838 10667 tablet_bootstrap.cc:654] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Neither blocks nor log segments found. Creating new log.
I20260927 06:59:33.284957 10666 log.cc:824] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:33.284871 10664 raft_consensus.cc:348] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.285043 10664 raft_consensus.cc:374] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:33.285087 10664 raft_consensus.cc:729] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da37e7478e5e432a96ee5564f010d2fd, State: Initialized, Role: FOLLOWER
I20260927 06:59:33.285126 10667 log.cc:824] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Log is configured to *not* fsync() on all Append() calls
I20260927 06:59:33.285238 10664 consensus_queue.cc:260] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.285494 10664 ts_tablet_manager.cc:1428] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:33.285691 10366 heartbeater.cc:503] Master 127.3.57.62:41287 was elected leader, sending a full tablet report...
I20260927 06:59:33.285776 10666 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: No bootstrap required, opened a new log
I20260927 06:59:33.285781 10667 tablet_bootstrap.cc:492] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: No bootstrap required, opened a new log
I20260927 06:59:33.285833 10666 ts_tablet_manager.cc:1397] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:33.285832 10667 ts_tablet_manager.cc:1397] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 06:59:33.287829 10666 raft_consensus.cc:348] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.287987 10666 raft_consensus.cc:374] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:33.288019 10666 raft_consensus.cc:729] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d9609bdbcc443abbea3255ff647e8c, State: Initialized, Role: FOLLOWER
I20260927 06:59:33.288138 10666 consensus_queue.cc:260] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.288237 10667 raft_consensus.cc:348] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.288381 10666 ts_tablet_manager.cc:1428] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 06:59:33.288383 10667 raft_consensus.cc:374] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 06:59:33.288416 10667 raft_consensus.cc:729] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa73dc4a1b664debaa59feb5eaa7e65b, State: Initialized, Role: FOLLOWER
I20260927 06:59:33.288475 10503 heartbeater.cc:503] Master 127.3.57.62:41287 was elected leader, sending a full tablet report...
I20260927 06:59:33.288527 10667 consensus_queue.cc:260] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.288754 10667 ts_tablet_manager.cc:1428] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 06:59:33.288868 10639 heartbeater.cc:503] Master 127.3.57.62:41287 was elected leader, sending a full tablet report...
I20260927 06:59:33.351537 10671 raft_consensus.cc:482] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 06:59:33.351729 10671 raft_consensus.cc:504] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.352185 10671 leader_election.cc:288] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683), aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:33.355089 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.352328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39683 (local address 127.3.57.2:47387)
0927 06:59:33.352421 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.352435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.352483 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.352640 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.352641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.352644 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:33.352721 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.352723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.353467 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.353468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.354193 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.354195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.354842 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.354923 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.354926 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.354970 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.355001 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 06:59:33.355123 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.352403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:47387 (local address 127.3.57.1:39683)
0927 06:59:33.352456 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.352458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.352474 (+ 16us) server_negotiation.cc:407] Connection header received
0927 06:59:33.352565 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.352566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.352577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.352632 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:33.352802 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.353376 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354284 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.354762 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.354854 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.354995 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.355052 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 06:59:33.355190 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.352415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:40043 (local address 127.3.57.3:44451)
0927 06:59:33.352461 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.352463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.352473 (+ 10us) server_negotiation.cc:407] Connection header received
0927 06:59:33.352557 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.352558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.352570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.352663 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 06:59:33.352846 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.353324 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354431 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.354776 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.354874 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.355069 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.355118 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 06:59:33.355201 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.352371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:44451 (local address 127.3.57.2:40043)
0927 06:59:33.352424 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:33.352435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 06:59:33.352476 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 06:59:33.352692 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 06:59:33.352693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 06:59:33.352695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 06:59:33.352765 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.352767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.353409 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.353411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.354342 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 06:59:33.354345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.354885 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.354886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 06:59:33.354989 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.354993 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 06:59:33.355054 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 06:59:33.355094 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":56}
I20260927 06:59:33.355491 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f52a6ec0e12487abdced2d1ad827b99" candidate_uuid: "45d9609bdbcc443abbea3255ff647e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da37e7478e5e432a96ee5564f010d2fd" is_pre_election: true
I20260927 06:59:33.355687 10319 raft_consensus.cc:2454] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9609bdbcc443abbea3255ff647e8c in term 0.
I20260927 06:59:33.355993 10390 leader_election.cc:302] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9609bdbcc443abbea3255ff647e8c, da37e7478e5e432a96ee5564f010d2fd; no voters:
I20260927 06:59:33.356158 10671 raft_consensus.cc:2781] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 06:59:33.356207 10671 raft_consensus.cc:482] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 06:59:33.356230 10671 raft_consensus.cc:3037] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:33.356841 10592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f52a6ec0e12487abdced2d1ad827b99" candidate_uuid: "45d9609bdbcc443abbea3255ff647e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" is_pre_election: true
I20260927 06:59:33.357002 10592 raft_consensus.cc:2454] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d9609bdbcc443abbea3255ff647e8c in term 0.
I20260927 06:59:33.357288 10671 raft_consensus.cc:504] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.357431 10671 leader_election.cc:288] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [CANDIDATE]: Term 1 election: Requested vote from peers da37e7478e5e432a96ee5564f010d2fd (127.3.57.1:39683), aa73dc4a1b664debaa59feb5eaa7e65b (127.3.57.3:44451)
I20260927 06:59:33.357568 10319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f52a6ec0e12487abdced2d1ad827b99" candidate_uuid: "45d9609bdbcc443abbea3255ff647e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da37e7478e5e432a96ee5564f010d2fd"
I20260927 06:59:33.357618 10592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f52a6ec0e12487abdced2d1ad827b99" candidate_uuid: "45d9609bdbcc443abbea3255ff647e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b"
I20260927 06:59:33.357672 10319 raft_consensus.cc:3037] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:33.357682 10592 raft_consensus.cc:3037] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 0 FOLLOWER]: Advancing to term 1
I20260927 06:59:33.358369 10592 raft_consensus.cc:2454] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9609bdbcc443abbea3255ff647e8c in term 1.
I20260927 06:59:33.358475 10319 raft_consensus.cc:2454] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d9609bdbcc443abbea3255ff647e8c in term 1.
I20260927 06:59:33.358534 10392 leader_election.cc:302] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d9609bdbcc443abbea3255ff647e8c, aa73dc4a1b664debaa59feb5eaa7e65b; no voters:
I20260927 06:59:33.358634 10671 raft_consensus.cc:2781] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 06:59:33.358809 10671 raft_consensus.cc:686] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 1 LEADER]: Becoming Leader. State: Replica: 45d9609bdbcc443abbea3255ff647e8c, State: Running, Role: LEADER
I20260927 06:59:33.358924 10671 consensus_queue.cc:237] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } }
I20260927 06:59:33.359901 9401 catalog_manager.cc:5495] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d9609bdbcc443abbea3255ff647e8c (127.3.57.2). New cstate: current_term: 1 leader_uuid: "45d9609bdbcc443abbea3255ff647e8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d9609bdbcc443abbea3255ff647e8c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 42509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 } health_report { overall_health: UNKNOWN } } }
W20260927 06:59:33.361477 10505 tablet.cc:2369] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 06:59:33.371284 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.368994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.3.57.2:42509)
0927 06:59:33.369050 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.369052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.369060 (+ 8us) server_negotiation.cc:407] Connection header received
0927 06:59:33.369130 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.369131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.369148 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.369230 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 06:59:33.369331 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.369792 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.370539 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.370739 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.370826 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.370983 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 06:59:33.371083 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 06:59:33.371171 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.371191 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.371210 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 06:59:33.372985 10592 raft_consensus.cc:1261] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 1 FOLLOWER]: Refusing update from remote peer 45d9609bdbcc443abbea3255ff647e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:33.373090 10319 raft_consensus.cc:1261] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 1 FOLLOWER]: Refusing update from remote peer 45d9609bdbcc443abbea3255ff647e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 06:59:33.373315 10673 consensus_queue.cc:1035] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa73dc4a1b664debaa59feb5eaa7e65b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:33.373435 10671 consensus_queue.cc:1035] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da37e7478e5e432a96ee5564f010d2fd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 06:59:33.376072 10676 mvcc.cc:205] Tried to move back new op lower bound from 7333856761333043200 to 7333856761279307776. Current Snapshot: MvccSnapshot[applied={T|T < 7333856761333043200}]
I20260927 06:59:33.379096 9401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:33.379186 9401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 06:59:33.380590 9401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57f2e6c4243c434d9b1d25ff9653e653: Sending DeleteTablet for 3 replicas of tablet 2f52a6ec0e12487abdced2d1ad827b99
I20260927 06:59:33.380887 10436 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:33 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:58102
I20260927 06:59:33.380959 10572 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:33 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33492
I20260927 06:59:33.381314 10686 tablet_replica.cc:330] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: stopping tablet replica
I20260927 06:59:33.381464 10686 raft_consensus.cc:2229] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 1 LEADER]: Raft consensus shutting down.
I20260927 06:59:33.381659 10686 raft_consensus.cc:2258] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:33.381662 10299 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f52a6ec0e12487abdced2d1ad827b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 06:59:33 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35380
I20260927 06:59:33.381868 10689 tablet_replica.cc:330] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: stopping tablet replica
I20260927 06:59:33.381971 10689 raft_consensus.cc:2229] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:33.382074 10689 raft_consensus.cc:2258] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:33.382076 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9371
I20260927 06:59:33.382356 10686 ts_tablet_manager.cc:1905] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:33.383107 10689 ts_tablet_manager.cc:1905] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 06:59:33.384261 10686 ts_tablet_manager.cc:1918] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:33.384397 10686 log.cc:1196] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/2f52a6ec0e12487abdced2d1ad827b99
I20260927 06:59:33.384725 10686 ts_tablet_manager.cc:1939] T 2f52a6ec0e12487abdced2d1ad827b99 P 45d9609bdbcc443abbea3255ff647e8c: Deleting consensus metadata
I20260927 06:59:33.385033 10689 ts_tablet_manager.cc:1918] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:33.385107 10689 log.cc:1196] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/2f52a6ec0e12487abdced2d1ad827b99
I20260927 06:59:33.385375 10689 ts_tablet_manager.cc:1939] T 2f52a6ec0e12487abdced2d1ad827b99 P da37e7478e5e432a96ee5564f010d2fd: Deleting consensus metadata
I20260927 06:59:33.387915 10687 tablet_replica.cc:330] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: stopping tablet replica
I20260927 06:59:33.388176 10687 raft_consensus.cc:2229] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 06:59:33.388377 10687 raft_consensus.cc:2258] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 06:59:33.388908 10687 ts_tablet_manager.cc:1905] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 06:59:33.389415 10253 connection.cc:535] server connection from 127.0.0.1:35380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 06:59:33.389601 10392 connection.cc:535] server connection from 127.0.0.1:58102 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 06:59:33.390923 10687 ts_tablet_manager.cc:1918] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 06:59:33.391053 10687 log.cc:1196] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/2f52a6ec0e12487abdced2d1ad827b99
I20260927 06:59:33.391381 10687 ts_tablet_manager.cc:1939] T 2f52a6ec0e12487abdced2d1ad827b99 P aa73dc4a1b664debaa59feb5eaa7e65b: Deleting consensus metadata
W20260927 06:59:33.391788 10526 connection.cc:441] server connection from 127.0.0.1:33492 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33492 (request call id 1) could send its response
I20260927 06:59:33.395460 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.381533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.3.57.60:46671)
0927 06:59:33.381577 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.381579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.387752 (+ 6173us) server_negotiation.cc:407] Connection header received
0927 06:59:33.387831 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.387832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.387846 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.387897 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.388839 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.389549 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.390721 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.391011 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.391059 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.392841 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.392843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.392843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.392851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.394350 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.394707 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.394708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.394709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.394763 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.394965 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.394966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.394966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.395086 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.395188 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.395300 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.395370 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 06:59:33.399380 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.391860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.3.57.60:46671)
0927 06:59:33.391917 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.391919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.392323 (+ 404us) server_negotiation.cc:407] Connection header received
0927 06:59:33.392445 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.392446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.392478 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.392508 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.393168 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.393885 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.394912 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.395208 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.395245 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.396738 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.396740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.396740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.396747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.398291 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.398633 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.398634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.398635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.398687 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.398893 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.398894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.398895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.398998 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.399096 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.399190 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.399267 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 06:59:33.403373 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:33.392162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.3.57.61:42163)
0927 06:59:33.392209 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 06:59:33.392211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 06:59:33.396164 (+ 3953us) server_negotiation.cc:407] Connection header received
0927 06:59:33.396325 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 06:59:33.396327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 06:59:33.396340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 06:59:33.396389 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 06:59:33.396816 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.397438 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.399215 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 06:59:33.399513 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 06:59:33.399685 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 06:59:33.400990 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 06:59:33.400991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 06:59:33.400992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 06:59:33.400998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 06:59:33.402369 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.402736 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.402737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.402737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.402785 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 06:59:33.402977 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 06:59:33.402977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 06:59:33.402977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 06:59:33.403068 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 06:59:33.403134 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 06:59:33.403256 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 06:59:33.403314 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 06:59:34.377538 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.377265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:58331)
0927 06:59:34.377394 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:34.377484 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 06:59:34.377740 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.377474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:33875)
0927 06:59:34.377620 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:34.377691 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
W20260927 06:59:34.378044 10639 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:41287 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
W20260927 06:59:34.378288 10366 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:41287 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:34.378571 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.378376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:49327)
0927 06:59:34.378471 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:34.378528 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
W20260927 06:59:34.378827 10503 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:41287 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:34.881003 10700 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:34.881124 10700 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:34.881356 10700 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:34.881666 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.881475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37598)
0927 06:59:34.881585 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:34.881630 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
W20260927 06:59:34.881790 9985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:34.882138 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:34.882043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:37600)
0927 06:59:34.882091 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:34.882115 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 06:59:34.882227 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:34.882280 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:34.882369 10700 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:35.378808 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:35.378529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:39241)
0927 06:59:35.378631 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:35.378751 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260927 06:59:35.378877 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:35.378606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:49179)
0927 06:59:35.378705 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:35.378817 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260927 06:59:35.379626 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:35.379134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:41797)
0927 06:59:35.379512 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:35.379573 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":49,"threads_started":1}
I20260927 06:59:36.379812 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:36.379494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:48323)
0927 06:59:36.379709 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:36.379754 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260927 06:59:36.380250 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:36.379800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:41693)
0927 06:59:36.380173 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:36.380206 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":64,"threads_started":1}
I20260927 06:59:36.380474 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:36.380088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:52149)
0927 06:59:36.380366 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:36.380429 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":44,"threads_started":1}
I20260927 06:59:37.010625 10709 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:37.010720 10709 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:37.010962 10709 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:37.011283 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:37.011105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:34540)
0927 06:59:37.011210 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:37.011249 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260927 06:59:37.011700 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:37.011631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:34548)
0927 06:59:37.011661 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:37.011683 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 06:59:37.011804 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:37.011861 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:37.011942 10709 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:37.380517 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:37.380287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:54875)
0927 06:59:37.380417 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:37.380456 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260927 06:59:37.381001 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:37.380823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:40181)
0927 06:59:37.380932 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:37.380960 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260927 06:59:37.381129 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:37.380672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:38099)
0927 06:59:37.381053 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:37.381086 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":45,"threads_started":1}
I20260927 06:59:38.381546 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:38.381235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:59995)
0927 06:59:38.381442 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:38.381486 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
I20260927 06:59:38.381811 10716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:38.381436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:47209)
0927 06:59:38.381722 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:38.381761 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":70,"threads_started":1}
I20260927 06:59:38.382076 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:38.381814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:54087)
0927 06:59:38.382003 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:38.382036 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
I20260927 06:59:39.382606 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:39.382205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:57183)
0927 06:59:39.382496 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:39.382540 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":61,"threads_started":1}
I20260927 06:59:39.382725 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:39.382344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:34011)
0927 06:59:39.382640 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:39.382676 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":69,"threads_started":1}
I20260927 06:59:39.382964 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:39.382530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:47911)
0927 06:59:39.382888 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:39.382923 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":49,"threads_started":1}
I20260927 06:59:40.058629 10721 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:40.058754 10721 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:40.058941 10721 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:40.059309 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.059126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:34852)
0927 06:59:40.059236 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:40.059271 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
W20260927 06:59:40.059396 9985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260927 06:59:40.059758 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.059672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:34866)
0927 06:59:40.059716 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:40.059741 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 06:59:40.059834 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:40.059891 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:40.059979 10721 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:40.383721 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.383479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:47649)
0927 06:59:40.383642 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:40.383670 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":39,"threads_started":1}
I20260927 06:59:40.383822 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.383334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:49673)
0927 06:59:40.383620 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:40.383765 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":64,"threads_started":1}
I20260927 06:59:40.383946 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:40.383157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:47221)
0927 06:59:40.383467 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:40.383512 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":62,"threads_started":1}
I20260927 06:59:41.384752 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:41.384419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:34119)
0927 06:59:41.384644 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:41.384688 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260927 06:59:41.384862 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:41.384612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:56341)
0927 06:59:41.384794 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:41.384821 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":47,"threads_started":1}
I20260927 06:59:41.384862 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:41.384463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:59919)
0927 06:59:41.384769 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:41.384808 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":57,"threads_started":1}
I20260927 06:59:42.385610 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:42.385323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:36183)
0927 06:59:42.385519 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:42.385559 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
I20260927 06:59:42.385610 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:42.385294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:59597)
0927 06:59:42.385532 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:42.385559 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":60,"threads_started":1}
I20260927 06:59:42.385649 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:42.385458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:53565)
0927 06:59:42.385582 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:42.385612 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260927 06:59:43.386248 10733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.385960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:50235)
0927 06:59:43.386159 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:43.386197 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260927 06:59:43.386248 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.386060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:56809)
0927 06:59:43.386159 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:43.386197 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260927 06:59:43.386440 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.386060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:35135)
0927 06:59:43.386359 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:43.386391 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":65,"threads_started":1}
I20260927 06:59:43.683102 10736 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:43.683208 10736 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:43.683331 10736 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:43.683715 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.683510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:35126)
0927 06:59:43.683633 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:43.683674 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260927 06:59:43.684144 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:43.684060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:35140)
0927 06:59:43.684103 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:43.684127 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 06:59:43.684222 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:43.684274 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:43.684333 10736 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:44.387027 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:44.386650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:46599)
0927 06:59:44.386934 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:44.386975 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":66,"threads_started":1}
I20260927 06:59:44.387027 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:44.386821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:55381)
0927 06:59:44.386930 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:44.386973 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":44,"threads_started":1}
I20260927 06:59:44.387171 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:44.386675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:42389)
0927 06:59:44.387089 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:44.387124 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":57,"threads_started":1}
I20260927 06:59:45.387766 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:45.387495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:42213)
0927 06:59:45.387666 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:45.387708 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260927 06:59:45.387908 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:45.387522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:43601)
0927 06:59:45.387769 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:45.387807 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
I20260927 06:59:45.388100 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:45.387522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:49995)
0927 06:59:45.387958 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:45.388059 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260927 06:59:46.388602 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:46.388315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:54619)
0927 06:59:46.388506 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:46.388546 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":114,"threads_started":1}
I20260927 06:59:46.388602 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:46.388368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:42249)
0927 06:59:46.388534 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:46.388567 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260927 06:59:46.388775 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:46.388568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:42551)
0927 06:59:46.388686 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:46.388722 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":44,"threads_started":1}
I20260927 06:59:47.389364 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:47.389085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:59241)
0927 06:59:47.389256 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:47.389310 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":62,"threads_started":1}
I20260927 06:59:47.389357 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:47.389186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:39029)
0927 06:59:47.389275 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:47.389313 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":50,"threads_started":1}
I20260927 06:59:47.389642 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:47.389476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:43117)
0927 06:59:47.389570 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:47.389603 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
I20260927 06:59:48.389973 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:48.389759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:47585)
0927 06:59:48.389876 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:48.389914 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260927 06:59:48.390112 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:48.389894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:36837)
0927 06:59:48.390030 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:48.390061 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260927 06:59:48.390233 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:48.390036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:44147)
0927 06:59:48.390144 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:48.390189 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260927 06:59:49.390678 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:49.390455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:38645)
0927 06:59:49.390595 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:49.390634 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260927 06:59:49.390808 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:49.390608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:43333)
0927 06:59:49.390732 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:49.390764 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":42,"threads_started":1}
I20260927 06:59:49.390808 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:49.390656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:50147)
0927 06:59:49.390745 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:49.390769 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":39,"threads_started":1}
I20260927 06:59:50.232308 10757 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:50.232419 10757 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:50.232596 10757 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:50.232985 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:50.232798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:50682)
0927 06:59:50.232912 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:50.232946 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
W20260927 06:59:50.233070 9985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260927 06:59:50.233419 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:50.233328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:50686)
0927 06:59:50.233378 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:50.233402 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 06:59:50.233500 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:50.233574 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:50.233657 10757 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:50.391441 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:50.391195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:40609)
0927 06:59:50.391339 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:50.391384 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260927 06:59:50.391480 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:50.391307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:52171)
0927 06:59:50.391402 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:50.391442 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":42,"threads_started":1}
I20260927 06:59:50.391480 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:50.391269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:53083)
0927 06:59:50.391402 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:50.391434 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260927 06:59:51.392072 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:51.391889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:46841)
0927 06:59:51.391975 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:51.392023 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":41,"threads_started":1}
I20260927 06:59:51.392072 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:51.391858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:36877)
0927 06:59:51.391996 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:51.392026 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260927 06:59:51.392073 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:51.391839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:42505)
0927 06:59:51.391975 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:51.392019 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260927 06:59:52.392709 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:52.392469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:47535)
0927 06:59:52.392613 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:52.392655 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260927 06:59:52.392709 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:52.392516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:59447)
0927 06:59:52.392637 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:52.392665 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260927 06:59:52.392735 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:52.392570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:51425)
0927 06:59:52.392657 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:52.392696 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
I20260927 06:59:52.697973 10769 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:52.698099 10769 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:52.698253 10769 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:52.698607 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:52.698403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:50818)
0927 06:59:52.698524 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:52.698568 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":47,"threads_started":1}
I20260927 06:59:52.699070 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:52.698970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:50826)
0927 06:59:52.699020 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:52.699047 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 06:59:52.699186 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:52.699272 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:52.699357 10769 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:53.393337 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:53.393115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:44031)
0927 06:59:53.393263 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:53.393291 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":53,"threads_started":1}
I20260927 06:59:53.393337 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:53.393095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:51755)
0927 06:59:53.393242 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:53.393282 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260927 06:59:53.393369 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:53.393185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:33935)
0927 06:59:53.393295 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:53.393333 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":40,"threads_started":1}
I20260927 06:59:54.393971 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.393735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:41625)
0927 06:59:54.393886 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.393921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260927 06:59:54.393971 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.393730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:38123)
0927 06:59:54.393876 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.393921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260927 06:59:54.394114 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:54.393843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:56353)
0927 06:59:54.393936 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:54.394000 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
I20260927 06:59:55.394521 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.394319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:52057)
0927 06:59:55.394432 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.394469 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260927 06:59:55.394563 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.394355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:34249)
0927 06:59:55.394480 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.394513 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":52,"threads_started":1}
I20260927 06:59:55.394640 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:55.394475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:33043)
0927 06:59:55.394557 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:55.394600 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":39,"threads_started":1}
I20260927 06:59:56.395218 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.394965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:43049)
0927 06:59:56.395122 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:56.395161 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260927 06:59:56.395218 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.394967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:59055)
0927 06:59:56.395122 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:56.395161 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260927 06:59:56.395332 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:56.394970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:44203)
0927 06:59:56.395223 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:56.395255 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
I20260927 06:59:57.395911 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.395652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:35763)
0927 06:59:57.395803 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:57.395852 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260927 06:59:57.395959 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.395652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:53591)
0927 06:59:57.395872 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:57.395911 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":67,"threads_started":1}
I20260927 06:59:57.396010 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:57.395826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:53315)
0927 06:59:57.395942 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:57.395970 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260927 06:59:58.057386 10787 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57f2e6c4243c434d9b1d25ff9653e653)
I20260927 06:59:58.057497 10787 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57f2e6c4243c434d9b1d25ff9653e653" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41287 } } peers { permanent_uuid: "a555d7cefc394026a632b51a9d409f13" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 42163 } attrs { promote: false } } peers { permanent_uuid: "4ddcf2a7cfb540e2a1a76c2d57744ab2" member_type: NON_VOTER last_known_addr { host: "127.3.57.60" port: 46671 } attrs { promote: true } }
I20260927 06:59:58.057679 10787 leader_election.cc:288] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287)
I20260927 06:59:58.058058 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:58.057849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:53730)
0927 06:59:58.057979 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:58.058021 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
W20260927 06:59:58.058139 9985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260927 06:59:58.058485 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:58.058396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.0.0.1:53738)
0927 06:59:58.058443 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:58.058468 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 06:59:58.058563 9985 leader_election.cc:334] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57f2e6c4243c434d9b1d25ff9653e653 (127.3.57.62:41287): Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
I20260927 06:59:58.058617 9985 leader_election.cc:302] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a555d7cefc394026a632b51a9d409f13; no voters: 57f2e6c4243c434d9b1d25ff9653e653
I20260927 06:59:58.058702 10787 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a555d7cefc394026a632b51a9d409f13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 06:59:58.396623 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:58.396381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:54397)
0927 06:59:58.396522 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:58.396563 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260927 06:59:58.396690 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:58.396421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:40233)
0927 06:59:58.396603 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:58.396640 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":40,"threads_started":1}
I20260927 06:59:58.396723 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:58.396470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:44055)
0927 06:59:58.396653 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:58.396683 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260927 06:59:59.397358 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.397132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:53813)
0927 06:59:59.397262 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.397303 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260927 06:59:59.397375 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.397132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:44697)
0927 06:59:59.397313 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.397340 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":57,"threads_started":1}
I20260927 06:59:59.397380 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 06:59:59.397104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:47777)
0927 06:59:59.397242 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 06:59:59.397342 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 07:00:00.397953 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.397749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:43089)
0927 07:00:00.397861 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.397902 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":38,"threads_started":1}
I20260927 07:00:00.397959 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.397721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:59181)
0927 07:00:00.397864 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.397902 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260927 07:00:00.397953 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:00.397721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:41481)
0927 07:00:00.397874 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:00.397903 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
I20260927 07:00:01.398708 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.398453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:52715)
0927 07:00:01.398605 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:01.398650 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260927 07:00:01.398757 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.398489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:54939)
0927 07:00:01.398689 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:01.398715 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260927 07:00:01.398834 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:01.398599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:54819)
0927 07:00:01.398745 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:01.398782 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":49,"threads_started":1}
I20260927 07:00:02.399521 10803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.399266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:52325)
0927 07:00:02.399394 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:02.399449 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":45,"threads_started":1}
I20260927 07:00:02.399586 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.399182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:36393)
0927 07:00:02.399445 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:02.399484 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":68,"threads_started":1}
I20260927 07:00:02.399763 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:02.399525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:53331)
0927 07:00:02.399686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:02.399724 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260927 07:00:03.400307 10805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:03.400064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.3:49823)
0927 07:00:03.400208 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:03.400251 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260927 07:00:03.400411 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:03.400079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.1:42541)
0927 07:00:03.400340 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:03.400372 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":86,"threads_started":1}
I20260927 07:00:03.400478 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:03.400249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41287 (local address 127.3.57.2:45799)
0927 07:00:03.400344 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:03.400378 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:41287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.3.57.62:41287,127.3.57.61:42163,127.3.57.60:46671, num_attempts: 1)
I20260927 07:00:03.572732 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10147
I20260927 07:00:03.577806 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10373
I20260927 07:00:03.583021 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10509
I20260927 07:00:03.587263 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 9867
I20260927 07:00:03.591823 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10054
2026-09-27T07:00:03Z chronyd exiting
I20260927 07:00:03.607604 3300 test_util.cc:175] -----------------------------------------------
I20260927 07:00:03.607657 3300 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492341973781-3300-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33015 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:00:03 dist-test-slave-mk3h krb5kdc[10810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:00:03 dist-test-slave-mk3h krb5kdc[10810](info): set up 2 sockets
Sep 27 07:00:03 dist-test-slave-mk3h krb5kdc[10810](info): commencing operation
krb5kdc: starting...
W20260927 07:00:04.643860 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:00:04 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab.
Entry for principal oryx/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab.
I20260927 07:00:04.764005 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40767
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab
--principal=oryx/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:04.844487 10823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:04.844835 10823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:04.844954 10823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:04.847177 10823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:04.847270 10823 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:00:04.847299 10823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:04.847333 10823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:04.847366 10823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:04.847397 10823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:04.849161 10823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.62.keytab
--principal=oryx/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:04.849440 10823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:04.849680 10823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:04.855158 10831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:04.855158 10828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:04.855212 10823 server_base.cc:1034] running on GCE node
W20260927 07:00:04.855139 10829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:04.855659 10823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:04.855944 10823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:04.857085 10823 hybrid_clock.cc:648] HybridClock initialized: now 1790492404857075 us; error 31 us; skew 500 ppm
Sep 27 07:00:04 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492404, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:04.859411 10823 init.cc:377] Logged in from keytab as oryx/127.3.57.62@KRBTEST.COM (short username oryx)
I20260927 07:00:04.859920 10823 webserver.cc:460] Webserver started at http://127.3.57.62:45133/ using document root <none> and password file <none>
I20260927 07:00:04.860141 10823 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:04.860219 10823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:04.860369 10823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:04.861426 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "63f276eb15484503950e3f88047579af"
format_stamp: "Formatted at 2026-09-27 07:00:04 on dist-test-slave-mk3h"
server_key: "c3b2691dd04c36c980fd16a7c2d54477"
server_key_iv: "351d79f203aba8ddcd3f4e4c17a1fd26"
server_key_version: "encryptionkey@0"
I20260927 07:00:04.861780 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "63f276eb15484503950e3f88047579af"
format_stamp: "Formatted at 2026-09-27 07:00:04 on dist-test-slave-mk3h"
server_key: "c3b2691dd04c36c980fd16a7c2d54477"
server_key_iv: "351d79f203aba8ddcd3f4e4c17a1fd26"
server_key_version: "encryptionkey@0"
I20260927 07:00:04.863052 10823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:00:04.863799 10838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:04.863973 10823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:04.864073 10823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "63f276eb15484503950e3f88047579af"
format_stamp: "Formatted at 2026-09-27 07:00:04 on dist-test-slave-mk3h"
server_key: "c3b2691dd04c36c980fd16a7c2d54477"
server_key_iv: "351d79f203aba8ddcd3f4e4c17a1fd26"
server_key_version: "encryptionkey@0"
I20260927 07:00:04.864151 10823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:04.876443 10823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:04.878170 10823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:04.878327 10823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:04.883370 10823 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:40767
I20260927 07:00:04.883410 10900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:40767 every 8 connection(s)
I20260927 07:00:04.883781 10823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:04.884378 10901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:04.886546 10901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Bootstrap starting.
I20260927 07:00:04.887135 10901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:04.887430 10901 log.cc:824] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:04.887814 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10823
I20260927 07:00:04.887921 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:00:04.888129 3300 external_mini_cluster.cc:1428] Setting key e9984337fa661ce3aad73c8de8ff6e5d
I20260927 07:00:04.888257 10901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: No bootstrap required, opened a new log
I20260927 07:00:04.890477 10901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } }
I20260927 07:00:04.890630 10901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:04.890694 10901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f276eb15484503950e3f88047579af, State: Initialized, Role: FOLLOWER
I20260927 07:00:04.890823 10901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } }
I20260927 07:00:04.890921 10901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:04.890964 10901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:04.891011 10901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:04.891497 10901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } }
I20260927 07:00:04.891634 10901 leader_election.cc:302] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63f276eb15484503950e3f88047579af; no voters:
I20260927 07:00:04.891788 10901 leader_election.cc:288] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:00:04.891832 10906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:04.891978 10906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [term 1 LEADER]: Becoming Leader. State: Replica: 63f276eb15484503950e3f88047579af, State: Running, Role: LEADER
I20260927 07:00:04.892052 10901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:04.892168 10906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } }
I20260927 07:00:04.892578 10908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63f276eb15484503950e3f88047579af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } } }
I20260927 07:00:04.892578 10906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63f276eb15484503950e3f88047579af. Latest consensus state: current_term: 1 leader_uuid: "63f276eb15484503950e3f88047579af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f276eb15484503950e3f88047579af" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40767 } } }
I20260927 07:00:04.892902 10906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:04.892793 10908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af [sys.catalog]: This master's current role is: LEADER
W20260927 07:00:04.893935 10921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:00:04.894018 10921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:00:04.894084 10915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:04.894229 10915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:04.895901 10915 catalog_manager.cc:1347] Generated new cluster ID: 7642c4cd32954bc09bc320f98696001e
I20260927 07:00:04.895951 10915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:00:04 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:04.901535 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:04.888674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.3.57.62:40767)
0927 07:00:04.888843 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:04.888858 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:04.888882 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:00:04.890438 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:04.890461 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:04.890517 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:04.890654 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:04.891355 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:04.892718 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:04.893855 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:04.894449 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:04.894551 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:04.897459 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:04.897463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:04.897464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:04.897475 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:04.899406 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:04.899881 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:04.899885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:04.899885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:04.900003 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:04.900235 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:04.900236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:04.900236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:04.900542 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:04.900691 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:04.901198 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 07:00:04.901333 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260927 07:00:04.903654 10915 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:04.904121 10915 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:04.910324 10915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Generated new TSK 0
I20260927 07:00:04.910504 10915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
I20260927 07:00:04.941702 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:40767
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:05.034157 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:05.034338 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:05.034363 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:05.035509 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:05.035605 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:05.035646 10927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:00:05.035672 10927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:05.037263 10927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:05.037494 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:05.037737 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:05.042570 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:05.042586 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:05.042630 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:05.042824 10927 server_base.cc:1034] running on GCE node
I20260927 07:00:05.043020 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:05.043228 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:05.044394 10927 hybrid_clock.cc:648] HybridClock initialized: now 1790492405044377 us; error 32 us; skew 500 ppm
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:05.046689 10927 init.cc:377] Logged in from keytab as oryx/127.3.57.1@KRBTEST.COM (short username oryx)
I20260927 07:00:05.047161 10927 webserver.cc:460] Webserver started at http://127.3.57.1:34213/ using document root <none> and password file <none>
I20260927 07:00:05.047400 10927 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:05.047466 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:05.047626 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:05.048614 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "6d92e35f6398455d8d45441ef83f4488"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "94eeeafdc0ecfd409b724ecb771d3d7a"
server_key_iv: "41a560a4eebc7d9eb3a906a001683f98"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.048962 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "6d92e35f6398455d8d45441ef83f4488"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "94eeeafdc0ecfd409b724ecb771d3d7a"
server_key_iv: "41a560a4eebc7d9eb3a906a001683f98"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.050216 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:05.050918 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.051111 10927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:05.051208 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "6d92e35f6398455d8d45441ef83f4488"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "94eeeafdc0ecfd409b724ecb771d3d7a"
server_key_iv: "41a560a4eebc7d9eb3a906a001683f98"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.051293 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:05.059944 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:05.061741 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:05.061898 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:05.062410 10927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:05.062471 10927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.062529 10927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:05.062559 10927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:05.069903 10927 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:33665
I20260927 07:00:05.070004 11057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:33665 every 8 connection(s)
I20260927 07:00:05.070322 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:00:05.072435 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.062760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:38413 (local address 127.3.57.62:40767)
0927 07:00:05.062810 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.062812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.063067 (+ 255us) server_negotiation.cc:407] Connection header received
0927 07:00:05.063779 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.063781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.063810 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.063864 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.064714 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.065248 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.066823 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.067104 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.067174 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.069753 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.069754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.069755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.069764 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.071347 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.071757 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.071758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.071758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.071806 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.071967 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.071967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.071967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.072053 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.072100 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.072306 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.072373 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:00:05.072569 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.062740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.1:38413)
0927 07:00:05.062912 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.062953 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.063653 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.063950 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.063954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.064315 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.064606 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.064613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.065353 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.065354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.066740 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.066743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.067204 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.067206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.067389 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.067901 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.067909 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.069640 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.071417 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.071421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.071422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.071702 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.071849 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.071849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.071849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.071930 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.072108 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.072113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.072215 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.072287 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.072393 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260927 07:00:05.073050 11058 heartbeater.cc:347] Connected to a master server at 127.3.57.62:40767
I20260927 07:00:05.073153 11058 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:05.073316 11058 heartbeater.cc:511] Master 127.3.57.62:40767 requested a full tablet report, sending...
I20260927 07:00:05.073849 10854 ts_manager.cc:194] Registered new tserver with Master: 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665)
I20260927 07:00:05.074774 10854 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:38413
I20260927 07:00:05.076757 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 10927
I20260927 07:00:05.076884 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:00:05.077106 3300 external_mini_cluster.cc:1428] Setting key bec4c0d7eac6d76ab15864e15d371750
I20260927 07:00:05.079931 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.074179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:34617 (local address 127.3.57.62:40767)
0927 07:00:05.074217 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.074218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.074329 (+ 111us) server_negotiation.cc:407] Connection header received
0927 07:00:05.074395 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.074396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.074403 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.074449 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.074777 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.075420 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.076152 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.076325 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.076375 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.077383 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.077384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.077385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.077391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.078805 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.079159 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.079160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.079160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.079218 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.079415 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.079415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.079416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.079520 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.079618 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.079769 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.079841 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:05.079945 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.074074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.1:34617)
0927 07:00:05.074222 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.074233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.074341 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.074455 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.074455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.074651 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.074707 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.074709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.075522 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.075523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.076085 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.076087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.076388 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.076388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.076433 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.076811 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.076817 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.077207 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.078903 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.078904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.078905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.079075 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.079282 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.079283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.079284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.079342 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.079612 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.079613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.079658 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.079744 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.079842 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
I20260927 07:00:05.101752 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:40767
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:05.180819 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:05.181005 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:05.181028 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:05.182083 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:05.182116 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:05.182135 11067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:00:05.182152 11067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:05.183539 11067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:05.183728 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:05.183879 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:05.188488 11075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:05.188524 11067 server_base.cc:1034] running on GCE node
W20260927 07:00:05.188488 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:05.188503 11073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:05.188882 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:05.189138 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:05.190303 11067 hybrid_clock.cc:648] HybridClock initialized: now 1790492405190300 us; error 35 us; skew 500 ppm
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:05.194119 11067 init.cc:377] Logged in from keytab as oryx/127.3.57.2@KRBTEST.COM (short username oryx)
I20260927 07:00:05.194635 11067 webserver.cc:460] Webserver started at http://127.3.57.2:41493/ using document root <none> and password file <none>
I20260927 07:00:05.194891 11067 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:05.194972 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:05.195132 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:05.196344 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "825ff8650ba4496ba71bac5599992a92"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "f9647f87a9c2277a4b196e90be5a6b18"
server_key_iv: "02ce4bb65661fe0db1b7d92d010dc090"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.196743 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "825ff8650ba4496ba71bac5599992a92"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "f9647f87a9c2277a4b196e90be5a6b18"
server_key_iv: "02ce4bb65661fe0db1b7d92d010dc090"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.198225 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:05.199047 11082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.199250 11067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:05.199352 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "825ff8650ba4496ba71bac5599992a92"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "f9647f87a9c2277a4b196e90be5a6b18"
server_key_iv: "02ce4bb65661fe0db1b7d92d010dc090"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.199432 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:05.215180 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:05.217456 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:05.217617 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:05.218077 11067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:05.218138 11067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.218176 11067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:05.218204 11067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:05.225186 11067 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:33323
I20260927 07:00:05.225250 11197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:33323 every 8 connection(s)
I20260927 07:00:05.225567 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:05.226132 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11067
I20260927 07:00:05.226455 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:00:05.226676 3300 external_mini_cluster.cc:1428] Setting key d34e55ad83e80d50613344ba94704132
I20260927 07:00:05.228317 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.218586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:59289 (local address 127.3.57.62:40767)
0927 07:00:05.218634 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.218635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.218803 (+ 168us) server_negotiation.cc:407] Connection header received
0927 07:00:05.219305 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.219306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.219318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.219373 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.220186 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.221019 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.222574 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.222929 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.223018 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.225381 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.225383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.225383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.225391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.226943 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.227389 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.227391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.227391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.227439 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.227718 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.227718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.227719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.227815 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.227868 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.228194 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.228256 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:00:05.228585 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.218574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.2:59289)
0927 07:00:05.218695 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.218725 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.219201 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.219438 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.219442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.219802 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.220092 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.220098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.221104 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.221105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.222465 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.222469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.223033 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.223034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.223174 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.223824 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.223834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.225300 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.227027 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.227033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.227035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.227308 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.227487 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.227488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.227488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.227652 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.227883 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.227889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.228058 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.228158 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.228322 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260927 07:00:05.229074 11198 heartbeater.cc:347] Connected to a master server at 127.3.57.62:40767
I20260927 07:00:05.229161 11198 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:05.229339 11198 heartbeater.cc:511] Master 127.3.57.62:40767 requested a full tablet report, sending...
I20260927 07:00:05.229682 10854 ts_manager.cc:194] Registered new tserver with Master: 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:05.230182 10854 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:59289
WARNING: no policy specified for oryx/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:00:05.235651 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.230022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.2:42447)
0927 07:00:05.230148 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.230159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.230211 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.230389 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.230390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.230591 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.230644 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.230645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.231276 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.231277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.231851 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.231853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.232230 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.232230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.232295 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.232720 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.232725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.232987 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.234560 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.234561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.234561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.234729 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.234929 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.234948 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.234948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.235005 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.235310 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.235311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.235350 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.235441 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.235534 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260927 07:00:05.235680 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.230149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:42447 (local address 127.3.57.62:40767)
0927 07:00:05.230186 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.230188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.230198 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:05.230317 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.230318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.230328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.230386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.230725 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.231179 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.231957 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.232157 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.232231 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.233078 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.233079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.233080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.233099 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.234454 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.234822 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.234823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.234823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.234875 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.235077 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.235078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.235078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.235217 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.235304 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.235465 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.235539 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
I20260927 07:00:05.251693 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:40767
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:05.330317 11207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:05.330520 11207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:05.330543 11207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:05.331614 11207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:05.331646 11207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:05.331665 11207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:00:05.331682 11207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:05.333070 11207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.57.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:05.333240 11207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:05.333457 11207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:05.338176 11212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:05.338207 11215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:05.338263 11213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:05.338303 11207 server_base.cc:1034] running on GCE node
I20260927 07:00:05.338608 11207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:05.338855 11207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:05.339993 11207 hybrid_clock.cc:648] HybridClock initialized: now 1790492405339977 us; error 35 us; skew 500 ppm
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:05.342413 11207 init.cc:377] Logged in from keytab as oryx/127.3.57.3@KRBTEST.COM (short username oryx)
I20260927 07:00:05.342779 11207 webserver.cc:460] Webserver started at http://127.3.57.3:43927/ using document root <none> and password file <none>
I20260927 07:00:05.343006 11207 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:05.343078 11207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:05.343227 11207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:05.344308 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "2c3cc51b9368478bb7297ef33810a9a3"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "5982835060e023a9362c3472aedd4120"
server_key_iv: "7edec2baca265b8042e5e9151f849365"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.344668 11207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "2c3cc51b9368478bb7297ef33810a9a3"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "5982835060e023a9362c3472aedd4120"
server_key_iv: "7edec2baca265b8042e5e9151f849365"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.345974 11207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:05.346673 11222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.346839 11207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:05.346935 11207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "2c3cc51b9368478bb7297ef33810a9a3"
format_stamp: "Formatted at 2026-09-27 07:00:05 on dist-test-slave-mk3h"
server_key: "5982835060e023a9362c3472aedd4120"
server_key_iv: "7edec2baca265b8042e5e9151f849365"
server_key_version: "encryptionkey@0"
I20260927 07:00:05.347012 11207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:05.370648 11207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:05.372968 11207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:05.373162 11207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:05.373783 11207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:05.373839 11207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.373872 11207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:05.373891 11207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.380507 11207 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:37631
I20260927 07:00:05.380573 11337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:37631 every 8 connection(s)
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:05.380880 11207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:00:05.383656 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.374203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:36789 (local address 127.3.57.62:40767)
0927 07:00:05.374256 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.374258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.374466 (+ 208us) server_negotiation.cc:407] Connection header received
0927 07:00:05.374922 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.374924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.374938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.375028 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.375864 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.376729 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.378332 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.378685 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.378777 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.381160 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.381163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.381163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.381171 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.382294 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.382677 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.382679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.382679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.382733 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.383007 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.383009 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.383009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.383125 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.383214 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.383470 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.383569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:00:05.383814 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.374189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.3:36789)
0927 07:00:05.374334 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.374370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.374834 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.375092 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.375096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.375438 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.375774 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.375780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.376821 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.376822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.378225 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.378229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.378784 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.378784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.378877 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.379495 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.379507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.381066 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.382388 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.382394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.382396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.382602 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.382824 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.382825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.382826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.382937 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.383204 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.383209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.383338 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.383446 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.383603 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260927 07:00:05.384276 11338 heartbeater.cc:347] Connected to a master server at 127.3.57.62:40767
I20260927 07:00:05.384370 11338 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:05.384531 11338 heartbeater.cc:511] Master 127.3.57.62:40767 requested a full tablet report, sending...
I20260927 07:00:05.384876 10855 ts_manager.cc:194] Registered new tserver with Master: 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631)
I20260927 07:00:05.385385 10855 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:36789
I20260927 07:00:05.385957 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11207
I20260927 07:00:05.386077 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:00:05.386276 3300 external_mini_cluster.cc:1428] Setting key 73a8a97a4aca09831c061e5884f76b0a
I20260927 07:00:05.387140 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:00:05.391166 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.385230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.3.57.3:56801)
0927 07:00:05.385339 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.385348 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.385403 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.385550 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.385551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.385724 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.385803 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.385805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.386667 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.386668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.387412 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.387414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.387768 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.387769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.387840 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.388168 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.388174 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.388539 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.390286 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.390287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.390287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.390450 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.390652 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.390653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.390653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.390701 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.390956 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.390957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.390987 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.391036 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.391102 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260927 07:00:05.391263 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.385289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:56801 (local address 127.3.57.62:40767)
0927 07:00:05.385323 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.385325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.385393 (+ 68us) server_negotiation.cc:407] Connection header received
0927 07:00:05.385441 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.385442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.385451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.385506 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.385909 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.386566 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.387485 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.387712 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.387764 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.388619 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.388620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.388620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.388626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.390207 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.390528 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.390529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.390530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.390589 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.390760 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.390761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.390761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.390896 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.390961 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.391059 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.391135 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:00:05.408951 10905 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:05.403762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.3.57.62:40767)
0927 07:00:05.403811 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.403813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.403823 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:05.403894 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.403895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.403905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.403978 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.404490 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.404959 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.405717 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.405916 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.406024 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.408867 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492405, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:05.416327 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.409649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.3.57.62:40767)
0927 07:00:05.409706 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.409708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.409741 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:00:05.409817 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.409818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.409826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.409880 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.410397 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.410849 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.411865 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.412175 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.412255 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.414129 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.414131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.414131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.414138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.415232 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.415590 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.415591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.415591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.415646 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.415863 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.415864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.415864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.415965 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.416076 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.416172 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.416236 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:00:05.417678 10855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:00:05.419251 10855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:05.425403 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:33665 (local address 127.0.0.1:35910)
0927 07:00:05.421618 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.421629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.421691 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.421999 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.422002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.422004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:05.422119 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.422122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.423010 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.423011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.424175 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.424178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.425099 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.425101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.425186 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.425200 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.425274 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.425327 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260927 07:00:05.425534 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:37631 (local address 127.0.0.1:42734)
0927 07:00:05.421696 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.421705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.421743 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.422184 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.422185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.422187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:05.422215 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.422216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.423626 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.423627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.424517 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.424519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.425258 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.425259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.425312 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.425315 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.425359 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.425389 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260927 07:00:05.425566 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.3.57.1:33665)
0927 07:00:05.421859 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.421862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.421870 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:05.421899 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.421901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.421912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.421996 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:05.422200 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.422903 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.424274 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.424995 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.425077 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.425148 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.425303 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.425347 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":110,"threads_started":1}
I20260927 07:00:05.425588 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.3.57.3:37631)
0927 07:00:05.421992 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.421997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.422005 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:05.422039 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.422041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.422055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.422130 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:05.422285 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.423373 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.424612 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.425093 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.425157 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.425226 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.425440 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.425480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":69,"threads_started":1}
I20260927 07:00:05.426040 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:33323 (local address 127.0.0.1:40004)
0927 07:00:05.421577 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.421615 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.421848 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.422069 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.422070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.422072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:05.422129 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.422131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.423462 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.423463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.424558 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.424560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.425840 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.425842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.425904 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.425908 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.425944 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.425979 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":40,"threads_started":1}
I20260927 07:00:05.426174 10992 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec (DEFAULT_TABLE table=test-table [id=e1fee37504dc4e319878d891f9028572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.426168 11272 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec (DEFAULT_TABLE table=test-table [id=e1fee37504dc4e319878d891f9028572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.426470 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.421424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.3.57.2:33323)
0927 07:00:05.421732 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.421737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.421749 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:00:05.421952 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.421955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.421971 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.422201 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:05.422260 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.423373 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.424646 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.425359 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.426208 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.426308 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.426335 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.426375 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":64,"threads_started":1}
I20260927 07:00:05.426555 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c7fcceb414579ad9f69f26549c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.426553 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c7fcceb414579ad9f69f26549c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.426995 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec (DEFAULT_TABLE table=test-table [id=e1fee37504dc4e319878d891f9028572]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.427301 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2c7fcceb414579ad9f69f26549c3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.429615 11366 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Bootstrap starting.
I20260927 07:00:05.430274 11366 tablet_bootstrap.cc:654] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.430578 11366 log.cc:824] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:05.430730 11367 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Bootstrap starting.
I20260927 07:00:05.431180 11369 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Bootstrap starting.
I20260927 07:00:05.431361 11367 tablet_bootstrap.cc:654] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.431708 11367 log.cc:824] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:05.431834 11369 tablet_bootstrap.cc:654] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.431857 11366 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: No bootstrap required, opened a new log
I20260927 07:00:05.431910 11366 ts_tablet_manager.cc:1397] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:00:05.432173 11369 log.cc:824] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:05.432425 11367 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: No bootstrap required, opened a new log
I20260927 07:00:05.432499 11367 ts_tablet_manager.cc:1397] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:00:05.432958 11369 tablet_bootstrap.cc:492] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: No bootstrap required, opened a new log
I20260927 07:00:05.433075 11369 ts_tablet_manager.cc:1397] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:00:05.433631 11366 raft_consensus.cc:348] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.433795 11366 raft_consensus.cc:374] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.433835 11366 raft_consensus.cc:729] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.433951 11366 consensus_queue.cc:260] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.434230 11366 ts_tablet_manager.cc:1428] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 07:00:05.434238 11338 heartbeater.cc:503] Master 127.3.57.62:40767 was elected leader, sending a full tablet report...
I20260927 07:00:05.434551 11367 raft_consensus.cc:348] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.434703 11367 raft_consensus.cc:374] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.434746 11367 raft_consensus.cc:729] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.434854 11367 consensus_queue.cc:260] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.435088 11367 ts_tablet_manager.cc:1428] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:00:05.435197 11058 heartbeater.cc:503] Master 127.3.57.62:40767 was elected leader, sending a full tablet report...
I20260927 07:00:05.435084 11369 raft_consensus.cc:348] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.435472 11369 raft_consensus.cc:374] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.435520 11369 raft_consensus.cc:729] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ff8650ba4496ba71bac5599992a92, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.435827 11369 consensus_queue.cc:260] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.436094 11198 heartbeater.cc:503] Master 127.3.57.62:40767 was elected leader, sending a full tablet report...
I20260927 07:00:05.436093 11369 ts_tablet_manager.cc:1428] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260927 07:00:05.476572 11199 tablet.cc:2369] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:00:05.571401 11059 tablet.cc:2369] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:00:05.631488 11339 tablet.cc:2369] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:00:05.774415 11373 raft_consensus.cc:482] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:05.774585 11373 raft_consensus.cc:504] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.774924 11373 leader_election.cc:288] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323), 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631)
I20260927 07:00:05.777938 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.775199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:48781 (local address 127.3.57.2:33323)
0927 07:00:05.775259 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.775261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.775296 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:00:05.775355 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.775356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.775368 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.775438 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:05.775635 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.776166 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777259 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.777606 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777686 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.777695 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.777822 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.777862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:00:05.777953 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.775435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:49051 (local address 127.3.57.3:37631)
0927 07:00:05.775481 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.775482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.775489 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:00:05.775512 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.775513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.775523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.775587 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:05.775764 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.776210 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777260 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.777606 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777686 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.777696 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.777849 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.777884 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:00:05.778012 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.775125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:33323 (local address 127.3.57.1:48781)
0927 07:00:05.775231 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.775243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.775294 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.775452 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.775453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.775455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:05.775568 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.775571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.776264 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.776265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.777143 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.777145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.777689 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.777755 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.777758 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.777828 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.777876 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":56,"thread_start_us":63,"threads_started":1}
I20260927 07:00:05.778023 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.775292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:37631 (local address 127.3.57.1:49051)
0927 07:00:05.775406 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.775417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.775461 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.775647 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.775648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.775650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:05.775697 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.775698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.776317 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.776318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.777142 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.777144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.777685 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.777686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.777753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.777757 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.777832 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.777870 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":2,"thread_start_us":43,"threads_started":1}
I20260927 07:00:05.778388 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c7fcceb414579ad9f69f26549c3ec" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92" is_pre_election: true
I20260927 07:00:05.778390 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c7fcceb414579ad9f69f26549c3ec" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3cc51b9368478bb7297ef33810a9a3" is_pre_election: true
I20260927 07:00:05.778533 11152 raft_consensus.cc:2454] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 0.
I20260927 07:00:05.778538 11283 raft_consensus.cc:2454] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 0.
I20260927 07:00:05.778741 10944 leader_election.cc:302] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:05.778857 11373 raft_consensus.cc:2781] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:05.778939 11373 raft_consensus.cc:482] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:05.778975 11373 raft_consensus.cc:3037] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.779701 11373 raft_consensus.cc:504] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.779829 11373 leader_election.cc:288] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 election: Requested vote from peers 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323), 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631)
I20260927 07:00:05.780018 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c7fcceb414579ad9f69f26549c3ec" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92"
I20260927 07:00:05.780045 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2c7fcceb414579ad9f69f26549c3ec" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3cc51b9368478bb7297ef33810a9a3"
I20260927 07:00:05.780107 11283 raft_consensus.cc:3037] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.780108 11152 raft_consensus.cc:3037] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.780920 11152 raft_consensus.cc:2454] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 1.
I20260927 07:00:05.780941 11283 raft_consensus.cc:2454] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 1.
I20260927 07:00:05.781114 10944 leader_election.cc:302] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d92e35f6398455d8d45441ef83f4488, 825ff8650ba4496ba71bac5599992a92; no voters:
I20260927 07:00:05.781200 11373 raft_consensus.cc:2781] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:05.781332 11373 raft_consensus.cc:686] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 1 LEADER]: Becoming Leader. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Running, Role: LEADER
I20260927 07:00:05.781440 11373 consensus_queue.cc:237] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } }
I20260927 07:00:05.782184 10855 catalog_manager.cc:5495] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "6d92e35f6398455d8d45441ef83f4488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } health_report { overall_health: HEALTHY } } }
I20260927 07:00:05.862668 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.860599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.3.57.1:33665)
0927 07:00:05.860660 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.860662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.860671 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:05.860713 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.860714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.860732 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.860787 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:05.860893 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.861364 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.862036 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.862202 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.862258 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.862372 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:05.862461 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:05.862515 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.862554 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.862569 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:00:05.864521 11152 raft_consensus.cc:1261] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Refusing update from remote peer 6d92e35f6398455d8d45441ef83f4488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:05.864521 11283 raft_consensus.cc:1261] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Refusing update from remote peer 6d92e35f6398455d8d45441ef83f4488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:05.864825 11373 consensus_queue.cc:1035] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:05.864979 11377 consensus_queue.cc:1035] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:05.867285 11382 mvcc.cc:205] Tried to move back new op lower bound from 7333856894418116608 to 7333856894081351680. Current Snapshot: MvccSnapshot[applied={T|T < 7333856894418116608}]
I20260927 07:00:05.867012 11380 mvcc.cc:205] Tried to move back new op lower bound from 7333856894418116608 to 7333856894081351680. Current Snapshot: MvccSnapshot[applied={T|T < 7333856894418116608}]
I20260927 07:00:05.870309 10854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:05.870417 10854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:05.871135 10854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Sending DeleteTablet for 3 replicas of tablet 8e2c7fcceb414579ad9f69f26549c3ec
I20260927 07:00:05.871481 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:05 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 07:00:05.871485 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:05 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:40004
I20260927 07:00:05.871732 11391 tablet_replica.cc:330] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: stopping tablet replica
I20260927 07:00:05.871708 11272 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2c7fcceb414579ad9f69f26549c3ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:05 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:42734
I20260927 07:00:05.871830 11391 raft_consensus.cc:2229] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:00:05.871954 11391 raft_consensus.cc:2258] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:05.871992 11390 tablet_replica.cc:330] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: stopping tablet replica
I20260927 07:00:05.872076 11390 raft_consensus.cc:2229] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:05.872104 10854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 07:00:05.872192 11392 tablet_replica.cc:330] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: stopping tablet replica
W20260927 07:00:05.872242 10854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:05.872274 11392 raft_consensus.cc:2229] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:05.872363 11392 raft_consensus.cc:2258] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:05.872488 11391 ts_tablet_manager.cc:1905] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:05.872511 11390 raft_consensus.cc:2258] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:05.872957 11392 ts_tablet_manager.cc:1905] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:05.873644 10992 tablet_service.cc:1467] Processing CreateTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (DEFAULT_TABLE table=test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.873750 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.874694 11391 ts_tablet_manager.cc:1918] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:05.874758 11391 log.cc:1196] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/8e2c7fcceb414579ad9f69f26549c3ec
I20260927 07:00:05.874984 11367 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Bootstrap starting.
I20260927 07:00:05.875027 11391 ts_tablet_manager.cc:1939] T 8e2c7fcceb414579ad9f69f26549c3ec P 6d92e35f6398455d8d45441ef83f4488: Deleting consensus metadata
I20260927 07:00:05.875099 11392 ts_tablet_manager.cc:1918] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:05.875166 11392 log.cc:1196] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/8e2c7fcceb414579ad9f69f26549c3ec
I20260927 07:00:05.875393 11367 tablet_bootstrap.cc:654] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.875438 11392 ts_tablet_manager.cc:1939] T 8e2c7fcceb414579ad9f69f26549c3ec P 2c3cc51b9368478bb7297ef33810a9a3: Deleting consensus metadata
I20260927 07:00:05.875746 10842 catalog_manager.cc:4841] TS 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665): tablet 8e2c7fcceb414579ad9f69f26549c3ec (table test-table [id=e1fee37504dc4e319878d891f9028572]) successfully deleted
I20260927 07:00:05.875999 11390 ts_tablet_manager.cc:1905] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:05.876076 10840 catalog_manager.cc:4841] TS 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631): tablet 8e2c7fcceb414579ad9f69f26549c3ec (table test-table [id=e1fee37504dc4e319878d891f9028572]) successfully deleted
I20260927 07:00:05.876155 11367 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: No bootstrap required, opened a new log
I20260927 07:00:05.876194 11367 ts_tablet_manager.cc:1397] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:05.876185 11272 tablet_service.cc:1467] Processing CreateTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (DEFAULT_TABLE table=test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.876279 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.876317 11367 raft_consensus.cc:348] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.876384 11367 raft_consensus.cc:374] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.876405 11367 raft_consensus.cc:729] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.876447 11367 consensus_queue.cc:260] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.876533 11367 ts_tablet_manager.cc:1428] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.877242 11366 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Bootstrap starting.
I20260927 07:00:05.876320 11132 tablet_service.cc:1467] Processing CreateTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (DEFAULT_TABLE table=test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:05.877614 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.877624 11366 tablet_bootstrap.cc:654] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.878196 11366 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: No bootstrap required, opened a new log
I20260927 07:00:05.878234 11366 ts_tablet_manager.cc:1397] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:05.878373 11366 raft_consensus.cc:348] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.878440 11366 raft_consensus.cc:374] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.878460 11366 raft_consensus.cc:729] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.878499 11366 consensus_queue.cc:260] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.878578 11366 ts_tablet_manager.cc:1428] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.878899 11369 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Bootstrap starting.
I20260927 07:00:05.879274 11369 tablet_bootstrap.cc:654] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.880617 11390 ts_tablet_manager.cc:1918] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:05.880617 11369 tablet_bootstrap.cc:492] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: No bootstrap required, opened a new log
I20260927 07:00:05.880709 11369 ts_tablet_manager.cc:1397] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:00:05.880712 11390 log.cc:1196] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/8e2c7fcceb414579ad9f69f26549c3ec
I20260927 07:00:05.880872 11369 raft_consensus.cc:348] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.880940 11369 raft_consensus.cc:374] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.880960 11369 raft_consensus.cc:729] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ff8650ba4496ba71bac5599992a92, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.881016 11369 consensus_queue.cc:260] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.881129 11369 ts_tablet_manager.cc:1428] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.881342 11390 ts_tablet_manager.cc:1939] T 8e2c7fcceb414579ad9f69f26549c3ec P 825ff8650ba4496ba71bac5599992a92: Deleting consensus metadata
I20260927 07:00:05.881656 10840 catalog_manager.cc:4841] TS 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323): tablet 8e2c7fcceb414579ad9f69f26549c3ec (table test-table [id=e1fee37504dc4e319878d891f9028572]) successfully deleted
I20260927 07:00:05.893488 11387 raft_consensus.cc:482] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:05.893548 11387 raft_consensus.cc:504] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.893697 11387 leader_election.cc:288] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323), 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631)
I20260927 07:00:05.893849 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c71a1c6f9b4ea9bd0b60ce5de537de" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3cc51b9368478bb7297ef33810a9a3" is_pre_election: true
I20260927 07:00:05.893862 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c71a1c6f9b4ea9bd0b60ce5de537de" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92" is_pre_election: true
I20260927 07:00:05.893926 11283 raft_consensus.cc:2454] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 0.
I20260927 07:00:05.893939 11152 raft_consensus.cc:2454] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 0.
I20260927 07:00:05.894059 10944 leader_election.cc:302] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:05.894168 11387 raft_consensus.cc:2781] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:05.894218 11387 raft_consensus.cc:482] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:05.894250 11387 raft_consensus.cc:3037] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.894820 11387 raft_consensus.cc:504] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.894948 11387 leader_election.cc:288] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 election: Requested vote from peers 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323), 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631)
I20260927 07:00:05.895072 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c71a1c6f9b4ea9bd0b60ce5de537de" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92"
I20260927 07:00:05.895088 11283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c71a1c6f9b4ea9bd0b60ce5de537de" candidate_uuid: "6d92e35f6398455d8d45441ef83f4488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3cc51b9368478bb7297ef33810a9a3"
I20260927 07:00:05.895143 11283 raft_consensus.cc:3037] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.895143 11152 raft_consensus.cc:3037] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:05.895838 11283 raft_consensus.cc:2454] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 1.
I20260927 07:00:05.895900 11152 raft_consensus.cc:2454] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d92e35f6398455d8d45441ef83f4488 in term 1.
I20260927 07:00:05.896006 10944 leader_election.cc:302] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:05.896118 11387 raft_consensus.cc:2781] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:05.896170 11387 raft_consensus.cc:686] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 1 LEADER]: Becoming Leader. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Running, Role: LEADER
I20260927 07:00:05.896216 11387 consensus_queue.cc:237] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.896808 10854 catalog_manager.cc:5495] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "6d92e35f6398455d8d45441ef83f4488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } health_report { overall_health: UNKNOWN } } }
Sep 27 07:00:05 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492404, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.62@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:05.927091 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.920152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40767 (local address 127.0.0.1:52440)
0927 07:00:05.920355 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:05.920367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:05.920452 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:05.920610 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:05.920610 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:05.920903 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:05.920943 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.920945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.921663 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.921664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.922172 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:05.922174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.922576 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.922576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:05.922634 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.923174 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:05.923178 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:05.924509 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:05.926037 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.926044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.926046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.926285 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.926534 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:05.926534 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:05.926534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:05.926588 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:05.926847 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:05.926852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:05.926892 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 07:00:05.926942 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:00:05.927020 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260927 07:00:05.927191 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:05.920214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.3.57.62:40767)
0927 07:00:05.920361 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:05.920363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:05.920436 (+ 73us) server_negotiation.cc:407] Connection header received
0927 07:00:05.920513 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:05.920513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:05.920524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:05.920607 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:05.921042 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.921591 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.922254 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:05.922452 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:05.922564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:05.924605 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:05.924607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:05.924607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:05.924614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:05.925938 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.926374 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.926375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.926376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.926453 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:05.926651 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:05.926651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:05.926652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:05.926785 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:05.926853 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:05.926997 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:00:05.927097 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260927 07:00:05.928964 10854 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:52440:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:00:05.929193 10854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:05.930630 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 8305a61056b642229da148f532ba82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55a7a6d38aa041c3af28464c14cb248d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:00:05.930648 10992 tablet_service.cc:1467] Processing CreateTablet for tablet 8305a61056b642229da148f532ba82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55a7a6d38aa041c3af28464c14cb248d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:00:05.930784 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8305a61056b642229da148f532ba82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.930790 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8305a61056b642229da148f532ba82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.931671 11272 tablet_service.cc:1467] Processing CreateTablet for tablet 8305a61056b642229da148f532ba82f2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=55a7a6d38aa041c3af28464c14cb248d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:00:05.931802 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8305a61056b642229da148f532ba82f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:05.931973 11369 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: Bootstrap starting.
I20260927 07:00:05.932358 11369 tablet_bootstrap.cc:654] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.933044 11369 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: No bootstrap required, opened a new log
I20260927 07:00:05.933105 11369 ts_tablet_manager.cc:1397] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:05.933260 11366 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: Bootstrap starting.
I20260927 07:00:05.933261 11369 raft_consensus.cc:348] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.933337 11369 raft_consensus.cc:374] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.933355 11369 raft_consensus.cc:729] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ff8650ba4496ba71bac5599992a92, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.933401 11369 consensus_queue.cc:260] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.933494 11369 ts_tablet_manager.cc:1428] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.933648 11366 tablet_bootstrap.cc:654] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.933636 11374 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:05.933722 11374 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:05.934085 11367 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: Bootstrap starting.
I20260927 07:00:05.934356 11367 tablet_bootstrap.cc:654] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:05.934465 11366 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: No bootstrap required, opened a new log
I20260927 07:00:05.934516 11366 ts_tablet_manager.cc:1397] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:05.934664 11366 raft_consensus.cc:348] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.934744 11366 raft_consensus.cc:374] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.934770 11366 raft_consensus.cc:729] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.934816 11366 consensus_queue.cc:260] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.934907 11366 ts_tablet_manager.cc:1428] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:05.935067 11367 tablet_bootstrap.cc:492] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: No bootstrap required, opened a new log
I20260927 07:00:05.935004 11372 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:05.935112 11367 ts_tablet_manager.cc:1397] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:05.935134 11372 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:05.935238 11367 raft_consensus.cc:348] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.935314 11367 raft_consensus.cc:374] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:05.935343 11367 raft_consensus.cc:729] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Initialized, Role: FOLLOWER
I20260927 07:00:05.935391 11367 consensus_queue.cc:260] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:05.935483 11367 ts_tablet_manager.cc:1428] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:05.935531 11387 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:05.935673 11387 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:06.184891 11372 raft_consensus.cc:482] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:06.185009 11372 raft_consensus.cc:504] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.185449 11372 leader_election.cc:288] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.188560 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.185871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:38451 (local address 127.3.57.2:33323)
0927 07:00:06.185912 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.185914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.185966 (+ 52us) server_negotiation.cc:407] Connection header received
0927 07:00:06.186039 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.186040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.186051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.186121 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:06.186297 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.186835 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.187864 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.188231 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.188285 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.188316 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.188449 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.188492 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:00:06.188581 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.185637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:33323 (local address 127.3.57.3:38451)
0927 07:00:06.185887 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.185898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.185988 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.186143 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.186144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.186146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:06.186212 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.186215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.186913 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.186914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.187784 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.187786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.188309 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.188310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.188380 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.188384 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.188426 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.188455 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260927 07:00:06.188648 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.185742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:54365 (local address 127.3.57.1:33665)
0927 07:00:06.185801 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.185803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.186035 (+ 232us) server_negotiation.cc:407] Connection header received
0927 07:00:06.186152 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.186153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.186162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.186215 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:06.186396 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.186885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.187891 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.188236 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.188322 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.188331 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.188528 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.188570 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 07:00:06.188836 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.185689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:33665 (local address 127.3.57.3:54365)
0927 07:00:06.185973 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.185983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.186062 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.186219 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.186224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.186225 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:06.186319 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.186321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.186976 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.186977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.187790 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.187792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.188341 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.188341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.188443 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.188446 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.188501 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.188535 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260927 07:00:06.188899 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8305a61056b642229da148f532ba82f2" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92" is_pre_election: true
I20260927 07:00:06.189004 11152 raft_consensus.cc:2454] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.189188 11224 leader_election.cc:302] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 825ff8650ba4496ba71bac5599992a92; no voters:
I20260927 07:00:06.189318 11372 raft_consensus.cc:2781] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:06.189234 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8305a61056b642229da148f532ba82f2" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488" is_pre_election: true
I20260927 07:00:06.189363 11372 raft_consensus.cc:482] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:06.189378 11372 raft_consensus.cc:3037] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.189393 11012 raft_consensus.cc:2454] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.190091 11372 raft_consensus.cc:504] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.190234 11372 leader_election.cc:288] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Requested vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.190374 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8305a61056b642229da148f532ba82f2" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92"
I20260927 07:00:06.190390 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8305a61056b642229da148f532ba82f2" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488"
I20260927 07:00:06.190465 11012 raft_consensus.cc:3037] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.190465 11152 raft_consensus.cc:3037] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.191191 11152 raft_consensus.cc:2454] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.191280 11012 raft_consensus.cc:2454] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.191359 11224 leader_election.cc:302] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 825ff8650ba4496ba71bac5599992a92; no voters:
I20260927 07:00:06.191464 11372 raft_consensus.cc:2781] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:06.191637 11372 raft_consensus.cc:686] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Running, Role: LEADER
I20260927 07:00:06.191740 11372 consensus_queue.cc:237] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.192051 11409 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: TxnStatusTablet state changed. Reason: New leader 2c3cc51b9368478bb7297ef33810a9a3. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.192144 11409 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:00:06.192368 11411 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:00:06.192461 11411 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:00:06.192451 10853 catalog_manager.cc:5495] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } health_report { overall_health: HEALTHY } } }
I20260927 07:00:06.219691 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.217021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:37631 (local address 127.0.0.1:45120)
0927 07:00:06.217127 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.217139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.217200 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.217407 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.217408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.217409 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:00:06.217464 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.217467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.218121 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.218122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.218706 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.218707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.219085 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.219086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.219148 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.219166 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.219473 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.219482 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.219563 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.219616 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 07:00:06.219662 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.217140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.3.57.3:37631)
0927 07:00:06.217185 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.217188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.217196 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:06.217279 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.217280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.217299 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.217399 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:06.217521 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.218033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.218799 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.219007 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.219087 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.219269 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.219377 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.219483 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.219536 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.219583 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:00:06.220849 11152 raft_consensus.cc:1261] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.220966 11012 raft_consensus.cc:1261] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.221109 11409 consensus_queue.cc:1035] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.221324 11372 consensus_queue.cc:1035] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.222237 11387 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: TxnStatusTablet state changed. Reason: New leader 2c3cc51b9368478bb7297ef33810a9a3. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.222342 11387 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:06.222396 11374 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: TxnStatusTablet state changed. Reason: New leader 2c3cc51b9368478bb7297ef33810a9a3. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.222472 11374 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:06.223327 11374 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.223345 11387 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.223408 11374 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 825ff8650ba4496ba71bac5599992a92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:06.223423 11387 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 6d92e35f6398455d8d45441ef83f4488: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:00:06.223945 11372 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.224040 11372 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:00:06.224017 11409 tablet_replica.cc:439] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } } }
I20260927 07:00:06.224090 11409 tablet_replica.cc:442] T 8305a61056b642229da148f532ba82f2 P 2c3cc51b9368478bb7297ef33810a9a3: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:00:06.230883 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.227653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:52095 (local address 127.3.57.3:37631)
0927 07:00:06.227694 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.227696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.228587 (+ 891us) server_negotiation.cc:407] Connection header received
0927 07:00:06.228654 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.228655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.228667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.228714 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:06.228884 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.229370 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.230196 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.230392 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.230484 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.230649 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.230688 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.230749 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.230801 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.230820 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:00:06.230926 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.227522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:37631 (local address 127.3.57.1:52095)
0927 07:00:06.228500 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.228510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.228602 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.228755 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.228756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.228758 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:00:06.228806 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.228808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.229453 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.229454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.230115 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.230117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.230486 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.230487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.230547 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.230570 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.230749 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.230751 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.230799 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.230845 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"thread_start_us":94,"threads_started":1}
I20260927 07:00:06.234431 11152 raft_consensus.cc:1261] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Refusing update from remote peer 6d92e35f6398455d8d45441ef83f4488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.234525 11283 raft_consensus.cc:1261] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Refusing update from remote peer 6d92e35f6398455d8d45441ef83f4488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.234755 11387 consensus_queue.cc:1035] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.235072 11373 consensus_queue.cc:1035] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.239878 11381 mvcc.cc:205] Tried to move back new op lower bound from 7333856895934058496 to 7333856894551396352. Current Snapshot: MvccSnapshot[applied={T|T < 7333856895934058496}]
I20260927 07:00:06.247262 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.244444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:33665 (local address 127.3.57.3:54619)
0927 07:00:06.244620 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.244631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.244690 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.244812 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.244812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.244813 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:00:06.244872 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.244874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.245720 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.245721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.246495 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.246497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.246911 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.246911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.246955 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.246966 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.247131 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.247133 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.247166 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.247197 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260927 07:00:06.247258 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.244479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:54619 (local address 127.3.57.1:33665)
0927 07:00:06.244519 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.244521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.244699 (+ 178us) server_negotiation.cc:407] Connection header received
0927 07:00:06.244738 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.244739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.244752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.244812 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:06.244909 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.245647 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.246570 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.246842 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.246909 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.247040 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.247075 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.247127 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.247173 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.247194 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:06.255785 10855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.255898 10855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.256490 10855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Sending DeleteTablet for 3 replicas of tablet d4c71a1c6f9b4ea9bd0b60ce5de537de
I20260927 07:00:06.256705 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 07:00:06.256805 11391 tablet_replica.cc:330] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: stopping tablet replica
I20260927 07:00:06.256811 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:40004
I20260927 07:00:06.256829 11272 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c71a1c6f9b4ea9bd0b60ce5de537de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:42734
I20260927 07:00:06.256875 11391 raft_consensus.cc:2229] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:00:06.256906 11390 tablet_replica.cc:330] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: stopping tablet replica
I20260927 07:00:06.256932 11391 raft_consensus.cc:2258] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.256961 11390 raft_consensus.cc:2229] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.256971 11392 tablet_replica.cc:330] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: stopping tablet replica
I20260927 07:00:06.257000 11390 raft_consensus.cc:2258] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.257026 11392 raft_consensus.cc:2229] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.257073 11392 raft_consensus.cc:2258] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.257334 11390 ts_tablet_manager.cc:1905] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.257418 11391 ts_tablet_manager.cc:1905] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.258920 11390 ts_tablet_manager.cc:1918] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:00:06.259006 11390 log.cc:1196] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/d4c71a1c6f9b4ea9bd0b60ce5de537de
I20260927 07:00:06.259029 11391 ts_tablet_manager.cc:1918] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:00:06.259086 11391 log.cc:1196] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/d4c71a1c6f9b4ea9bd0b60ce5de537de
I20260927 07:00:06.259263 11390 ts_tablet_manager.cc:1939] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 825ff8650ba4496ba71bac5599992a92: Deleting consensus metadata
I20260927 07:00:06.259316 11391 ts_tablet_manager.cc:1939] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 6d92e35f6398455d8d45441ef83f4488: Deleting consensus metadata
I20260927 07:00:06.259593 10840 catalog_manager.cc:4841] TS 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323): tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (table test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]) successfully deleted
I20260927 07:00:06.262533 10842 catalog_manager.cc:4841] TS 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665): tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (table test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]) successfully deleted
I20260927 07:00:06.263880 11392 ts_tablet_manager.cc:1905] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.265362 11392 ts_tablet_manager.cc:1918] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:00:06.265429 11392 log.cc:1196] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/d4c71a1c6f9b4ea9bd0b60ce5de537de
I20260927 07:00:06.265650 11392 ts_tablet_manager.cc:1939] T d4c71a1c6f9b4ea9bd0b60ce5de537de P 2c3cc51b9368478bb7297ef33810a9a3: Deleting consensus metadata
I20260927 07:00:06.265949 10840 catalog_manager.cc:4841] TS 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631): tablet d4c71a1c6f9b4ea9bd0b60ce5de537de (table test-table [id=5e5a0ef44fed4c9793fe52a5c866ccbc]) successfully deleted
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[10810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:00:06.282979 11403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:06.277188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.3.57.62:40767)
0927 07:00:06.277252 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.277254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.277349 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:00:06.277382 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.277383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.277396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.277464 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:06.277985 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.278735 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.279492 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.279828 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.279906 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.282914 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52442: BlockingRecv error: recv got EOF from 127.0.0.1:52442 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[10810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM
I20260927 07:00:06.290805 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.283741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.3.57.62:40767)
0927 07:00:06.283779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.283780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.283823 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:00:06.283927 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.283927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.283957 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.284023 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:06.284556 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.285092 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.285802 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.286003 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.286072 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.288152 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:06.288154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:06.288154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:06.288161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:06.289698 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:06.290047 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:06.290048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:06.290049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:06.290101 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:06.290301 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:06.290302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:06.290302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:06.290427 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:06.290526 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.290627 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.290706 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:06.292341 10855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:00:06.292531 10855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:06.293823 10992 tablet_service.cc:1467] Processing CreateTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 (DEFAULT_TABLE table=test-table [id=8a9aecc4f5f943e5930fd094c45aff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.293895 11272 tablet_service.cc:1467] Processing CreateTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 (DEFAULT_TABLE table=test-table [id=8a9aecc4f5f943e5930fd094c45aff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.293963 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa19d0a72b9d42dfb194e71268aab5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.294006 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa19d0a72b9d42dfb194e71268aab5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.294875 11366 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Bootstrap starting.
I20260927 07:00:06.295231 11366 tablet_bootstrap.cc:654] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.295320 11367 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Bootstrap starting.
I20260927 07:00:06.295356 11132 tablet_service.cc:1467] Processing CreateTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 (DEFAULT_TABLE table=test-table [id=8a9aecc4f5f943e5930fd094c45aff33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.295471 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa19d0a72b9d42dfb194e71268aab5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.295683 11367 tablet_bootstrap.cc:654] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.296227 11367 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: No bootstrap required, opened a new log
I20260927 07:00:06.296290 11367 ts_tablet_manager.cc:1397] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:06.296422 11367 raft_consensus.cc:348] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.296507 11367 raft_consensus.cc:374] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.296535 11367 raft_consensus.cc:729] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.296573 11367 consensus_queue.cc:260] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.296669 11367 ts_tablet_manager.cc:1428] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:06.296782 11369 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Bootstrap starting.
I20260927 07:00:06.297211 11369 tablet_bootstrap.cc:654] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.297855 11369 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: No bootstrap required, opened a new log
I20260927 07:00:06.297921 11369 ts_tablet_manager.cc:1397] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:00:06.298062 11369 raft_consensus.cc:348] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.298159 11369 raft_consensus.cc:374] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.298185 11369 raft_consensus.cc:729] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ff8650ba4496ba71bac5599992a92, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.298234 11369 consensus_queue.cc:260] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.298300 11366 tablet_bootstrap.cc:492] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: No bootstrap required, opened a new log
I20260927 07:00:06.298341 11369 ts_tablet_manager.cc:1428] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:06.298348 11366 ts_tablet_manager.cc:1397] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 07:00:06.298537 11366 raft_consensus.cc:348] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.298619 11366 raft_consensus.cc:374] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.298645 11366 raft_consensus.cc:729] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.298684 11366 consensus_queue.cc:260] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.298774 11366 ts_tablet_manager.cc:1428] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:06.304114 11416 raft_consensus.cc:482] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:06.304179 11416 raft_consensus.cc:504] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.304316 11416 leader_election.cc:288] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.304447 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa19d0a72b9d42dfb194e71268aab5b9" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488" is_pre_election: true
I20260927 07:00:06.304533 11012 raft_consensus.cc:2454] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.304626 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa19d0a72b9d42dfb194e71268aab5b9" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92" is_pre_election: true
I20260927 07:00:06.304703 11152 raft_consensus.cc:2454] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.304682 11226 leader_election.cc:302] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:06.304889 11416 raft_consensus.cc:2781] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:06.304939 11416 raft_consensus.cc:482] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:06.304960 11416 raft_consensus.cc:3037] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.305727 11416 raft_consensus.cc:504] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.305866 11416 leader_election.cc:288] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Requested vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.306062 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa19d0a72b9d42dfb194e71268aab5b9" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92"
I20260927 07:00:06.306088 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa19d0a72b9d42dfb194e71268aab5b9" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488"
I20260927 07:00:06.306131 11152 raft_consensus.cc:3037] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.306155 11012 raft_consensus.cc:3037] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.306782 11012 raft_consensus.cc:2454] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.306784 11152 raft_consensus.cc:2454] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.306945 11224 leader_election.cc:302] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 825ff8650ba4496ba71bac5599992a92; no voters:
I20260927 07:00:06.307051 11416 raft_consensus.cc:2781] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:06.307104 11416 raft_consensus.cc:686] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Running, Role: LEADER
I20260927 07:00:06.307143 11416 consensus_queue.cc:237] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.307659 10853 catalog_manager.cc:5495] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } health_report { overall_health: HEALTHY } } }
I20260927 07:00:06.314258 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.311929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.3.57.3:37631)
0927 07:00:06.311971 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.311973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.311982 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:06.312010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.312011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.312028 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.312095 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:06.312183 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.312866 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.313565 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.313868 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.313942 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.314045 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.314078 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.314126 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.314172 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.314188 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:00:06.314940 11152 raft_consensus.cc:1261] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.314992 11012 raft_consensus.cc:1261] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.315135 11416 consensus_queue.cc:1035] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.315258 11416 consensus_queue.cc:1035] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.318454 10855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.318539 10855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.319059 10855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Sending DeleteTablet for 3 replicas of tablet fa19d0a72b9d42dfb194e71268aab5b9
I20260927 07:00:06.319312 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 07:00:06.319401 11391 tablet_replica.cc:330] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: stopping tablet replica
I20260927 07:00:06.319401 11272 tablet_service.cc:1514] Processing DeleteTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:42734
I20260927 07:00:06.319414 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet fa19d0a72b9d42dfb194e71268aab5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:40004
I20260927 07:00:06.319445 11391 raft_consensus.cc:2229] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.319478 11392 tablet_replica.cc:330] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: stopping tablet replica
I20260927 07:00:06.319482 11391 raft_consensus.cc:2258] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.319516 11392 raft_consensus.cc:2229] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:00:06.319615 11392 raft_consensus.cc:2258] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.319692 11390 tablet_replica.cc:330] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: stopping tablet replica
I20260927 07:00:06.319751 11390 raft_consensus.cc:2229] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.319801 11390 raft_consensus.cc:2258] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.319841 11391 ts_tablet_manager.cc:1905] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.320111 11392 ts_tablet_manager.cc:1905] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.320120 10855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:00:06.320230 10855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:06.321287 11391 ts_tablet_manager.cc:1918] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:06.321323 11391 log.cc:1196] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/fa19d0a72b9d42dfb194e71268aab5b9
I20260927 07:00:06.321470 11391 ts_tablet_manager.cc:1939] T fa19d0a72b9d42dfb194e71268aab5b9 P 6d92e35f6398455d8d45441ef83f4488: Deleting consensus metadata
I20260927 07:00:06.321570 11392 ts_tablet_manager.cc:1918] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:06.321633 11392 log.cc:1196] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/fa19d0a72b9d42dfb194e71268aab5b9
I20260927 07:00:06.321794 10842 catalog_manager.cc:4841] TS 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665): tablet fa19d0a72b9d42dfb194e71268aab5b9 (table test-table [id=8a9aecc4f5f943e5930fd094c45aff33]) successfully deleted
I20260927 07:00:06.321828 11392 ts_tablet_manager.cc:1939] T fa19d0a72b9d42dfb194e71268aab5b9 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting consensus metadata
I20260927 07:00:06.322124 10840 catalog_manager.cc:4841] TS 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631): tablet fa19d0a72b9d42dfb194e71268aab5b9 (table test-table [id=8a9aecc4f5f943e5930fd094c45aff33]) successfully deleted
I20260927 07:00:06.322610 11390 ts_tablet_manager.cc:1905] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.322985 10992 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 (DEFAULT_TABLE table=test-table [id=2d0feca65055438297429e3a1576440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.323096 10992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c1fd98f6a2434283dec4cec67f1a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.323756 11390 ts_tablet_manager.cc:1918] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:06.323808 11390 log.cc:1196] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/fa19d0a72b9d42dfb194e71268aab5b9
I20260927 07:00:06.323859 11272 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 (DEFAULT_TABLE table=test-table [id=2d0feca65055438297429e3a1576440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.323956 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c1fd98f6a2434283dec4cec67f1a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.324018 11390 ts_tablet_manager.cc:1939] T fa19d0a72b9d42dfb194e71268aab5b9 P 825ff8650ba4496ba71bac5599992a92: Deleting consensus metadata
I20260927 07:00:06.324246 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 (DEFAULT_TABLE table=test-table [id=2d0feca65055438297429e3a1576440d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:06.324321 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c1fd98f6a2434283dec4cec67f1a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:06.324846 11367 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: Bootstrap starting.
I20260927 07:00:06.325189 11367 tablet_bootstrap.cc:654] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.325474 11366 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Bootstrap starting.
I20260927 07:00:06.325809 11367 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: No bootstrap required, opened a new log
I20260927 07:00:06.325855 11367 ts_tablet_manager.cc:1397] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:06.325892 11366 tablet_bootstrap.cc:654] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.325970 11367 raft_consensus.cc:348] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.326040 11367 raft_consensus.cc:374] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.326094 11367 raft_consensus.cc:729] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d92e35f6398455d8d45441ef83f4488, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.326141 11367 consensus_queue.cc:260] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.326232 10840 catalog_manager.cc:4841] TS 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323): tablet fa19d0a72b9d42dfb194e71268aab5b9 (table test-table [id=8a9aecc4f5f943e5930fd094c45aff33]) successfully deleted
I20260927 07:00:06.326247 11367 ts_tablet_manager.cc:1428] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:06.326273 11369 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Bootstrap starting.
I20260927 07:00:06.326915 11369 tablet_bootstrap.cc:654] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:06.327591 11369 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: No bootstrap required, opened a new log
I20260927 07:00:06.327600 11366 tablet_bootstrap.cc:492] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: No bootstrap required, opened a new log
I20260927 07:00:06.327643 11366 ts_tablet_manager.cc:1397] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:00:06.327644 11369 ts_tablet_manager.cc:1397] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:06.327795 11366 raft_consensus.cc:348] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.327805 11369 raft_consensus.cc:348] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.327868 11369 raft_consensus.cc:374] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.327870 11366 raft_consensus.cc:374] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:06.327889 11369 raft_consensus.cc:729] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ff8650ba4496ba71bac5599992a92, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.327894 11366 raft_consensus.cc:729] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Initialized, Role: FOLLOWER
I20260927 07:00:06.327935 11369 consensus_queue.cc:260] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.327961 11366 consensus_queue.cc:260] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.328015 11369 ts_tablet_manager.cc:1428] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:06.328058 11366 ts_tablet_manager.cc:1428] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:06.530864 11416 raft_consensus.cc:482] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:06.530979 11416 raft_consensus.cc:504] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.531206 11416 leader_election.cc:288] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.531430 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c1fd98f6a2434283dec4cec67f1a89" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488" is_pre_election: true
I20260927 07:00:06.531430 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c1fd98f6a2434283dec4cec67f1a89" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92" is_pre_election: true
I20260927 07:00:06.531523 11012 raft_consensus.cc:2454] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.531579 11152 raft_consensus.cc:2454] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 0.
I20260927 07:00:06.531714 11226 leader_election.cc:302] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:06.531831 11416 raft_consensus.cc:2781] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:06.531891 11416 raft_consensus.cc:482] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:06.531921 11416 raft_consensus.cc:3037] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.532612 11416 raft_consensus.cc:504] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.532759 11416 leader_election.cc:288] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Requested vote from peers 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665), 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323)
I20260927 07:00:06.532886 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c1fd98f6a2434283dec4cec67f1a89" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825ff8650ba4496ba71bac5599992a92"
I20260927 07:00:06.532889 11012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c1fd98f6a2434283dec4cec67f1a89" candidate_uuid: "2c3cc51b9368478bb7297ef33810a9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d92e35f6398455d8d45441ef83f4488"
I20260927 07:00:06.532955 11152 raft_consensus.cc:3037] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.532979 11012 raft_consensus.cc:3037] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:06.533567 11012 raft_consensus.cc:2454] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.533573 11152 raft_consensus.cc:2454] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3cc51b9368478bb7297ef33810a9a3 in term 1.
I20260927 07:00:06.533784 11226 leader_election.cc:302] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3cc51b9368478bb7297ef33810a9a3, 6d92e35f6398455d8d45441ef83f4488; no voters:
I20260927 07:00:06.533915 11416 raft_consensus.cc:2781] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:06.533967 11416 raft_consensus.cc:686] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2c3cc51b9368478bb7297ef33810a9a3, State: Running, Role: LEADER
I20260927 07:00:06.534024 11416 consensus_queue.cc:237] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } }
I20260927 07:00:06.534655 10855 catalog_manager.cc:5495] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "2c3cc51b9368478bb7297ef33810a9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3cc51b9368478bb7297ef33810a9a3" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 37631 } health_report { overall_health: HEALTHY } } }
I20260927 07:00:06.556022 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.554091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:55135 (local address 127.3.57.3:37631)
0927 07:00:06.554140 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:06.554141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:06.554148 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:00:06.554165 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:06.554166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:06.554174 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:06.554217 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:06.554361 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.554807 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.555456 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.555698 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.555749 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.555846 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.555867 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.555899 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:06.555931 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 07:00:06.555949 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":32}
I20260927 07:00:06.556030 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.553882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:37631 (local address 127.3.57.3:55135)
0927 07:00:06.553978 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.553990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:06.554038 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:06.554239 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:06.554240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:06.554241 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:00:06.554277 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.554278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.554861 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.554862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.555388 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:06.555390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:06.555761 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:06.555762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:06.555797 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:06.555802 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:00:06.555900 (+ 98us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:00:06.555900 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 07:00:06.555930 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 07:00:06.555965 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":1}
I20260927 07:00:06.557838 11152 raft_consensus.cc:1261] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.557829 11012 raft_consensus.cc:1261] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Refusing update from remote peer 2c3cc51b9368478bb7297ef33810a9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:06.558046 11372 consensus_queue.cc:1035] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825ff8650ba4496ba71bac5599992a92" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.558130 11409 consensus_queue.cc:1035] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d92e35f6398455d8d45441ef83f4488" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:06.567174 10853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.567272 10853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:06.567857 10853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63f276eb15484503950e3f88047579af: Sending DeleteTablet for 3 replicas of tablet 08c1fd98f6a2434283dec4cec67f1a89
I20260927 07:00:06.568063 10992 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35910
I20260927 07:00:06.568150 11391 tablet_replica.cc:330] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488: stopping tablet replica
I20260927 07:00:06.568145 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:40004
I20260927 07:00:06.568208 11391 raft_consensus.cc:2229] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.568228 11390 tablet_replica.cc:330] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: stopping tablet replica
I20260927 07:00:06.568243 11391 raft_consensus.cc:2258] T 08c1fd98f6a2434283dec4cec67f1a89 P 6d92e35f6398455d8d45441ef83f4488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.568272 11390 raft_consensus.cc:2229] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:06.568310 11390 raft_consensus.cc:2258] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.568496 11272 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c1fd98f6a2434283dec4cec67f1a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:06 UTC) from {username='oryx', principal='oryx/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:42734
I20260927 07:00:06.568524 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10927
I20260927 07:00:06.568593 11390 ts_tablet_manager.cc:1905] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.569160 11392 tablet_replica.cc:330] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: stopping tablet replica
I20260927 07:00:06.569221 11392 raft_consensus.cc:2229] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:00:06.569274 11392 raft_consensus.cc:2258] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:06.570137 11390 ts_tablet_manager.cc:1918] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:00:06.570175 11390 log.cc:1196] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/08c1fd98f6a2434283dec4cec67f1a89
I20260927 07:00:06.570325 11390 ts_tablet_manager.cc:1939] T 08c1fd98f6a2434283dec4cec67f1a89 P 825ff8650ba4496ba71bac5599992a92: Deleting consensus metadata
I20260927 07:00:06.570591 11392 ts_tablet_manager.cc:1905] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:06.571748 11392 ts_tablet_manager.cc:1918] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:00:06.571794 11392 log.cc:1196] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/08c1fd98f6a2434283dec4cec67f1a89
I20260927 07:00:06.572001 11392 ts_tablet_manager.cc:1939] T 08c1fd98f6a2434283dec4cec67f1a89 P 2c3cc51b9368478bb7297ef33810a9a3: Deleting consensus metadata
I20260927 07:00:06.572305 10840 catalog_manager.cc:4841] TS 825ff8650ba4496ba71bac5599992a92 (127.3.57.2:33323): tablet 08c1fd98f6a2434283dec4cec67f1a89 (table test-table [id=2d0feca65055438297429e3a1576440d]) successfully deleted
I20260927 07:00:06.572382 10840 catalog_manager.cc:4841] TS 2c3cc51b9368478bb7297ef33810a9a3 (127.3.57.3:37631): tablet 08c1fd98f6a2434283dec4cec67f1a89 (table test-table [id=2d0feca65055438297429e3a1576440d]) successfully deleted
W20260927 07:00:06.575177 10842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:33665 (error 108)
I20260927 07:00:06.575405 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11067
I20260927 07:00:06.576504 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:06.575591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:33665 (local address 127.0.0.1:48882)
0927 07:00:06.576415 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:06.576455 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:33665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":34,"threads_started":1}
W20260927 07:00:06.577056 10842 catalog_manager.cc:4584] TS 6d92e35f6398455d8d45441ef83f4488 (127.3.57.1:33665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08c1fd98f6a2434283dec4cec67f1a89: Network error: Client connection negotiation failed: client connection to 127.3.57.1:33665: connect: Connection refused (error 111)
I20260927 07:00:06.580266 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11207
I20260927 07:00:06.585976 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 10823
2026-09-27T07:00:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2998 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[11455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[11455](info): set up 2 sockets
Sep 27 07:00:06 dist-test-slave-mk3h krb5kdc[11455](info): commencing operation
krb5kdc: starting...
W20260927 07:00:07.607868 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:00:07 dist-test-slave-mk3h krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:00:07.717680 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36261
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:07.796767 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:07.796944 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:07.796965 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:07.798097 11468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:07.798135 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:07.798223 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:07.798250 11468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:07.798277 11468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:07.799783 11468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:07.800027 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:07.800251 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:00:07.804781 11468 server_base.cc:1034] running on GCE node
W20260927 07:00:07.804914 11473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:07.804906 11474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:07.804754 11476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:07.805254 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:07.805506 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:07.806643 11468 hybrid_clock.cc:648] HybridClock initialized: now 1790492407806622 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T07:00:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1227 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:00:07 dist-test-slave-mk3h krb5kdc[11481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:00:07 dist-test-slave-mk3h krb5kdc[11481](info): set up 2 sockets
Sep 27 07:00:07 dist-test-slave-mk3h krb5kdc[11481](info): commencing operation
krb5kdc: starting...
W20260927 07:00:08.827867 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:00:08 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:00:08.948248 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37561
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:09.026805 11494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.026999 11494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.027021 11494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.028184 11494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:09.028225 11494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.028239 11494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.028252 11494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:09.028304 11494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:09.029728 11494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.029968 11494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.030182 11494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.034876 11499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.034806 11500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.034823 11502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.034960 11494 server_base.cc:1034] running on GCE node
I20260927 07:00:09.035305 11494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.035638 11494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.036774 11494 hybrid_clock.cc:648] HybridClock initialized: now 1790492409036766 us; error 31 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.046341 11494 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:00:09.046802 11494 webserver.cc:460] Webserver started at http://127.3.57.62:37259/ using document root <none> and password file <none>
I20260927 07:00:09.046996 11494 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.047035 11494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.047122 11494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:09.048156 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "64e1980de75f42079d4bce2dfb4b0b64"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "940f61de78bf699fe922bc69159da2f0"
server_key_iv: "bcff5dc468a9f27401db4c63a6e126f6"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.048449 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "64e1980de75f42079d4bce2dfb4b0b64"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "940f61de78bf699fe922bc69159da2f0"
server_key_iv: "bcff5dc468a9f27401db4c63a6e126f6"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.049666 11494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:09.050314 11509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.050537 11494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.050647 11494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "64e1980de75f42079d4bce2dfb4b0b64"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "940f61de78bf699fe922bc69159da2f0"
server_key_iv: "bcff5dc468a9f27401db4c63a6e126f6"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.050721 11494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.071691 11494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.073895 11494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.074069 11494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.078156 11494 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37561
I20260927 07:00:09.078197 11561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37561 every 8 connection(s)
I20260927 07:00:09.078567 11494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:09.079413 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:09.081703 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Bootstrap starting.
I20260927 07:00:09.082226 11562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:09.082461 11562 log.cc:824] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:09.082885 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11494
I20260927 07:00:09.083019 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:00:09.083189 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: No bootstrap required, opened a new log
I20260927 07:00:09.083237 3300 external_mini_cluster.cc:1428] Setting key be254bf4529543b5c30896433fb788da
I20260927 07:00:09.085163 11562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.085323 11562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:09.085374 11562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e1980de75f42079d4bce2dfb4b0b64, State: Initialized, Role: FOLLOWER
I20260927 07:00:09.085503 11562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.085577 11562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:09.085598 11562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:09.085665 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:09.086179 11562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.086262 11562 leader_election.cc:302] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64e1980de75f42079d4bce2dfb4b0b64; no voters:
I20260927 07:00:09.086439 11562 leader_election.cc:288] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:00:09.086465 11567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:09.086585 11567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 LEADER]: Becoming Leader. State: Replica: 64e1980de75f42079d4bce2dfb4b0b64, State: Running, Role: LEADER
I20260927 07:00:09.086716 11562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:09.086725 11567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.087311 11568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64e1980de75f42079d4bce2dfb4b0b64. Latest consensus state: current_term: 1 leader_uuid: "64e1980de75f42079d4bce2dfb4b0b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } } }
I20260927 07:00:09.087474 11569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64e1980de75f42079d4bce2dfb4b0b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } } }
I20260927 07:00:09.087577 11568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:09.087670 11569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:09.087829 11576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:09.088341 11576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:09.089936 11576 catalog_manager.cc:1347] Generated new cluster ID: a1c7c248d4e74dfb866a2550198199d0
I20260927 07:00:09.089993 11576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:09.094705 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.083813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.3.57.62:37561)
0927 07:00:09.084010 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.084020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.084058 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:00:09.084526 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.084532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.084577 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.084723 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.085352 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.086865 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.088037 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.088712 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.088812 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.091353 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:09.091357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:09.091358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:09.091369 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:09.093199 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.093575 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.093578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.093578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.093668 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.093887 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.093888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.093889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.094120 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:09.094290 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:09.094367 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 07:00:09.094504 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260927 07:00:09.123016 11576 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:09.123595 11576 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:09.129508 11576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Generated new TSK 0
I20260927 07:00:09.129699 11576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:00:09.155367 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:09.234654 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.234861 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.234900 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.236114 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.236169 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.236213 11588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:09.237552 11588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.237794 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.238008 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.242853 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.242877 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.242993 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.242877 11588 server_base.cc:1034] running on GCE node
I20260927 07:00:09.243208 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.243467 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.244632 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790492409244614 us; error 34 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.247072 11588 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:00:09.247614 11588 webserver.cc:460] Webserver started at http://127.3.57.1:37785/ using document root <none> and password file <none>
I20260927 07:00:09.247849 11588 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.247931 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.248070 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:09.249066 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "c74ba60271db4a4ba7968da69098f48c"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "64b1f20d8e84722e75569214299f9ce9"
server_key_iv: "9d5e4c0ffecbd0d2106e18c2c6ce821a"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.249419 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "c74ba60271db4a4ba7968da69098f48c"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "64b1f20d8e84722e75569214299f9ce9"
server_key_iv: "9d5e4c0ffecbd0d2106e18c2c6ce821a"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.250718 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:00:09.251389 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.251590 11588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:09.251674 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "c74ba60271db4a4ba7968da69098f48c"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "64b1f20d8e84722e75569214299f9ce9"
server_key_iv: "9d5e4c0ffecbd0d2106e18c2c6ce821a"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.251760 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.261857 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.264055 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.264217 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.264420 11588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:09.264704 11588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:09.264757 11588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.264794 11588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:09.264822 11588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.270395 11588 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44137
I20260927 07:00:09.270468 11716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44137 every 8 connection(s)
I20260927 07:00:09.270735 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:09.279470 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.271369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:59571 (local address 127.3.57.62:37561)
0927 07:00:09.271409 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.271411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.271647 (+ 236us) server_negotiation.cc:407] Connection header received
0927 07:00:09.272104 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.272105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.272115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.272171 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.273023 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.273521 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.274590 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.274769 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.274817 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.276859 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:09.276861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:09.276861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:09.276867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:09.278226 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.278596 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.278597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.278597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.278645 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.278924 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.278925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.278925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.279051 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:09.279115 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:09.279339 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 07:00:09.279406 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:00:09.279650 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.271335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:59571)
0927 07:00:09.271493 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.271573 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.272018 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.272241 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.272244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.272776 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.272951 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.272955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.273626 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.273627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.274515 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.274517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.274839 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.274839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.274935 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.275369 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.275376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.276776 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:09.278307 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.278313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.278315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.278531 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.278727 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.278729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.278729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.278831 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.279120 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:09.279124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:09.279229 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 07:00:09.279309 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:00:09.279423 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260927 07:00:09.280167 11717 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37561
I20260927 07:00:09.280217 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11588
I20260927 07:00:09.280267 11717 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:09.280297 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:00:09.280457 11717 heartbeater.cc:511] Master 127.3.57.62:37561 requested a full tablet report, sending...
I20260927 07:00:09.280504 3300 external_mini_cluster.cc:1428] Setting key 4e9bd827a4ae58045f7cb83e03b5b6c3
I20260927 07:00:09.281011 11526 ts_manager.cc:194] Registered new tserver with Master: c74ba60271db4a4ba7968da69098f48c (127.3.57.1:44137)
I20260927 07:00:09.281679 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:59571
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:00:09.305943 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:09.384830 11722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.385027 11722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.385049 11722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.386194 11722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.386234 11722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.386260 11722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:09.388000 11722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.388224 11722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.388439 11722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.393342 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.393420 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.393461 11722 server_base.cc:1034] running on GCE node
W20260927 07:00:09.393438 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.393743 11722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.393981 11722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.395153 11722 hybrid_clock.cc:648] HybridClock initialized: now 1790492409395143 us; error 42 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.397504 11722 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:00:09.397980 11722 webserver.cc:460] Webserver started at http://127.3.57.2:45989/ using document root <none> and password file <none>
I20260927 07:00:09.398197 11722 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.398265 11722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.398408 11722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:09.399385 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "41ca6c60be4148e4ad094074a42751c7"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "79b8f5fa269759d8a9ba3271b595c0d0"
server_key_iv: "880d2e82969e3f06e8218b605cf1b9dd"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.399788 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "41ca6c60be4148e4ad094074a42751c7"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "79b8f5fa269759d8a9ba3271b595c0d0"
server_key_iv: "880d2e82969e3f06e8218b605cf1b9dd"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.401093 11722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.401839 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.402042 11722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:09.402141 11722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "41ca6c60be4148e4ad094074a42751c7"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "79b8f5fa269759d8a9ba3271b595c0d0"
server_key_iv: "880d2e82969e3f06e8218b605cf1b9dd"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.402222 11722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.421248 11722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.423102 11722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.423239 11722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.423475 11722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:09.423816 11722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:09.423871 11722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.423909 11722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:09.423961 11722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.429790 11722 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:37259
I20260927 07:00:09.429863 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:37259 every 8 connection(s)
I20260927 07:00:09.430181 11722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:09.430470 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11722
I20260927 07:00:09.430598 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:00:09.430833 3300 external_mini_cluster.cc:1428] Setting key 5392dfd00cbd73f28390185b9fbfeafa
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:00:09.439507 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.431023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:58133 (local address 127.3.57.62:37561)
0927 07:00:09.431079 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.431081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.431338 (+ 257us) server_negotiation.cc:407] Connection header received
0927 07:00:09.431936 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.431937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.431949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.432008 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.432928 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.433420 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.434396 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.434594 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.434699 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.436791 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:09.436792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:09.436793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:09.436800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:09.438125 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.438511 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.438512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.438513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.438572 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.438848 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.438849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.438850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.438959 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:09.439037 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:09.439334 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 07:00:09.439413 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:00:09.439769 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.431021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:58133)
0927 07:00:09.431196 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.431246 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.431829 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.432091 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.432095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.432643 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.432826 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.432831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.433514 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.433515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.434302 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.434304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.434706 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.434707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.434827 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.435338 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.435346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.436696 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:09.438209 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.438216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.438218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.438436 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.438641 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.438642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.438643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.438766 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.439028 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:09.439034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:09.439173 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 07:00:09.439307 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 07:00:09.439468 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260927 07:00:09.440210 11851 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37561
I20260927 07:00:09.440289 11851 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:09.440469 11851 heartbeater.cc:511] Master 127.3.57.62:37561 requested a full tablet report, sending...
I20260927 07:00:09.440785 11526 ts_manager.cc:194] Registered new tserver with Master: 41ca6c60be4148e4ad094074a42751c7 (127.3.57.2:37259)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
I20260927 07:00:09.441326 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:58133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:00:09.464035 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:09.548338 11856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.548569 11856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.548606 11856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.549780 11856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.549845 11856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.549889 11856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:09.551357 11856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.551630 11856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.551854 11856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.557153 11861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.557156 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.557111 11864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.557328 11856 server_base.cc:1034] running on GCE node
I20260927 07:00:09.557574 11856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.557813 11856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.558997 11856 hybrid_clock.cc:648] HybridClock initialized: now 1790492409558984 us; error 39 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.561745 11856 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:00:09.562222 11856 webserver.cc:460] Webserver started at http://127.3.57.3:37895/ using document root <none> and password file <none>
I20260927 07:00:09.562470 11856 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.562546 11856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.562705 11856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:09.563899 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "ae2fa522335e40b2b6acb61b2331b396"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "bd2281a6d2037c0d3e8e767555a6923e"
server_key_iv: "688b6dcffabaacac45c91cca01377817"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.564285 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "ae2fa522335e40b2b6acb61b2331b396"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "bd2281a6d2037c0d3e8e767555a6923e"
server_key_iv: "688b6dcffabaacac45c91cca01377817"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.565659 11856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:09.566448 11871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.566648 11856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.566758 11856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "ae2fa522335e40b2b6acb61b2331b396"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "bd2281a6d2037c0d3e8e767555a6923e"
server_key_iv: "688b6dcffabaacac45c91cca01377817"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.566835 11856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.584389 11856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.586323 11856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.586519 11856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.586759 11856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:09.587083 11856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:09.587141 11856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.587174 11856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:09.587198 11856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.593551 11856 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:37995
I20260927 07:00:09.593585 11984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:37995 every 8 connection(s)
I20260927 07:00:09.593910 11856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:00:09.599041 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11856
I20260927 07:00:09.599159 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:00:09.599354 3300 external_mini_cluster.cc:1428] Setting key 9708ab8cf829562714a45c5f7f8cb814
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:09.602658 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.594562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:54983 (local address 127.3.57.62:37561)
0927 07:00:09.594637 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.594640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.594799 (+ 159us) server_negotiation.cc:407] Connection header received
0927 07:00:09.595276 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.595277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.595288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.595339 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.596347 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.596827 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.597820 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.597997 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.598045 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.600332 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:09.600333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:09.600333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:09.600342 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:09.601555 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.601913 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.601913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.601914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.601966 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:09.602162 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:09.602162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:09.602162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:09.602268 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:09.602321 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:09.602518 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 07:00:09.602593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:00:09.602805 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.594527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:54983)
0927 07:00:09.594679 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.594715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.595193 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.595395 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.595398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.596067 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.596264 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.596270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.596910 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.596911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.597750 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.597752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.598068 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.598068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.598167 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.598595 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.598604 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.600224 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:09.601624 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.601629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.601631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.601845 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.602021 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:09.602021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:09.602022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:09.602108 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:09.602321 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:09.602325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:09.602428 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 07:00:09.602505 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:00:09.602611 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":122,"threads_started":1}
I20260927 07:00:09.603260 11985 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37561
I20260927 07:00:09.603371 11985 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:09.603602 11985 heartbeater.cc:511] Master 127.3.57.62:37561 requested a full tablet report, sending...
I20260927 07:00:09.603957 11526 ts_manager.cc:194] Registered new tserver with Master: ae2fa522335e40b2b6acb61b2331b396 (127.3.57.3:37995)
I20260927 07:00:09.604281 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:54983
I20260927 07:00:09.610984 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
Entry for principal oryx/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab.
WARNING: no policy specified for oryx/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
Entry for principal oryx/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab.
WARNING: no policy specified for oryx/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
Entry for principal oryx/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab.
I20260927 07:00:09.679956 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11588
I20260927 07:00:09.684478 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11722
I20260927 07:00:09.688875 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11856
I20260927 07:00:09.693186 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11494
I20260927 07:00:09.698271 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37561
--webserver_interface=127.3.57.62
--webserver_port=37259
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:09.779614 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.779835 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.779870 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.781162 11994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:09.781255 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.781284 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.781314 11994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:09.781338 11994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:09.782797 11994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.783061 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.783268 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.787912 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.788080 11994 server_base.cc:1034] running on GCE node
W20260927 07:00:09.787909 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.787921 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.788429 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.788710 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.789875 11994 hybrid_clock.cc:648] HybridClock initialized: now 1790492409789860 us; error 32 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.792269 11994 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:00:09.792689 11994 webserver.cc:460] Webserver started at http://127.3.57.62:37259/ using document root <none> and password file <none>
I20260927 07:00:09.792900 11994 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.792974 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.794312 11994 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.794955 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.795118 11994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.795212 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "64e1980de75f42079d4bce2dfb4b0b64"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "940f61de78bf699fe922bc69159da2f0"
server_key_iv: "bcff5dc468a9f27401db4c63a6e126f6"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.795543 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.801717 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.803869 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.804018 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.807821 11994 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37561
I20260927 07:00:09.807871 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37561 every 8 connection(s)
I20260927 07:00:09.808224 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:09.809512 12062 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:00:09.810151 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Bootstrap starting.
I20260927 07:00:09.812475 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 11994
I20260927 07:00:09.812819 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:44137
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:37561
--webserver_port=37785
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:00:09.818636 12062 log.cc:824] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:09.820333 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:00:09.820626 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Bootstrap complete.
I20260927 07:00:09.822788 12062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.822994 12062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e1980de75f42079d4bce2dfb4b0b64, State: Initialized, Role: FOLLOWER
I20260927 07:00:09.823153 12062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.823221 12062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:09.823277 12062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:09.823323 12062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:00:09.824313 12062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.824429 12062 leader_election.cc:302] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64e1980de75f42079d4bce2dfb4b0b64; no voters:
I20260927 07:00:09.824630 12062 leader_election.cc:288] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:00:09.824661 12066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:00:09.824766 12066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [term 2 LEADER]: Becoming Leader. State: Replica: 64e1980de75f42079d4bce2dfb4b0b64, State: Running, Role: LEADER
I20260927 07:00:09.824848 12066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } }
I20260927 07:00:09.824885 12062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:09.825098 12067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "64e1980de75f42079d4bce2dfb4b0b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } } }
I20260927 07:00:09.825179 12067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:09.825387 12068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64e1980de75f42079d4bce2dfb4b0b64. Latest consensus state: current_term: 2 leader_uuid: "64e1980de75f42079d4bce2dfb4b0b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1980de75f42079d4bce2dfb4b0b64" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37561 } } }
I20260927 07:00:09.825479 12068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:09.825605 12070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:09.826048 12070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:09.826164 12070 catalog_manager.cc:1259] Loaded cluster ID: a1c7c248d4e74dfb866a2550198199d0
I20260927 07:00:09.826424 12070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:00:09.827615 12070 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:09.827777 12070 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 64e1980de75f42079d4bce2dfb4b0b64: Loaded TSK: 0
I20260927 07:00:09.827987 12070 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:00:09.896966 12064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:09.897210 12064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:09.897238 12064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:09.898638 12064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:09.898684 12064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:09.898712 12064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:09.900319 12064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.1.keytab
--principal=oryx/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:44137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=37785
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:09.900578 12064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:09.900808 12064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:09.905984 12089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.906157 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:09.906056 12091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:09.906289 12064 server_base.cc:1034] running on GCE node
I20260927 07:00:09.906550 12064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:09.906816 12064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:09.907994 12064 hybrid_clock.cc:648] HybridClock initialized: now 1790492409907976 us; error 31 us; skew 500 ppm
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492409, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:09.910790 12064 init.cc:377] Logged in from keytab as oryx/127.3.57.1@KRBTEST.COM (short username oryx)
I20260927 07:00:09.911259 12064 webserver.cc:460] Webserver started at http://127.3.57.1:37785/ using document root <none> and password file <none>
I20260927 07:00:09.911501 12064 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:09.911607 12064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:09.912977 12064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.913584 12098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.913757 12064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:09.913856 12064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "c74ba60271db4a4ba7968da69098f48c"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "64b1f20d8e84722e75569214299f9ce9"
server_key_iv: "9d5e4c0ffecbd0d2106e18c2c6ce821a"
server_key_version: "encryptionkey@0"
I20260927 07:00:09.914172 12064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:09.922719 12064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:09.925004 12064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:09.925153 12064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:09.925415 12064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:09.925733 12064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:09.925781 12064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.925820 12064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:09.925848 12064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:09.932096 12064 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44137
I20260927 07:00:09.932147 12211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44137 every 8 connection(s)
I20260927 07:00:09.932444 12064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:00:09.937736 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12064
I20260927 07:00:09.938020 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:37259
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:37561
--webserver_port=45989
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:09.941419 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.933051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:38219)
0927 07:00:09.933239 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.933283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.933868 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.934440 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.934443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.934984 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.935146 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.935149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.936700 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.936701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.937428 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.937430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.938402 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.938404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.938551 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.939111 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.939121 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.941295 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
W20260927 07:00:09.941576 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:09.933176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:38219 (local address 127.3.57.62:37561)
0927 07:00:09.934064 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.934072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.934104 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:00:09.934215 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.934221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.934270 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.934402 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.935264 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.936597 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.937511 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.938304 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.938414 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.941415 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:38219: BlockingRecv error: recv got EOF from 127.3.57.1:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":69,"threads_started":1}
W20260927 07:00:09.941644 12212 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37561 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.57.62:37561: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:09.946167 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.942015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:42757)
0927 07:00:09.942075 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.942086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.942122 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.942310 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.942310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.942420 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.942457 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.942458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.943074 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.943075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.943653 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.943655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.943983 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.943984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.944052 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.944454 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.944458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.946087 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 07:00:09.946224 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:09.942045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:42757 (local address 127.3.57.62:37561)
0927 07:00:09.942099 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.942101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.942111 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:09.942183 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.942184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.942193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.942269 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.942534 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.942983 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.943740 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.943912 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.943982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.946163 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:42757: BlockingRecv error: recv got EOF from 127.3.57.1:42757 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:09 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:09.950791 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:09.946664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:51199)
0927 07:00:09.946772 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:09.946786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:09.946828 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:09.946964 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:09.946965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:09.947086 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:09.947111 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.947112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.947770 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.947771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.948260 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:09.948261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.948626 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.948627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:09.948682 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.949102 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:09.949107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:09.950712 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 07:00:09.950838 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:09.946723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:51199 (local address 127.3.57.62:37561)
0927 07:00:09.946771 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:09.946773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:09.946815 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:00:09.946883 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:09.946884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:09.946894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:09.946944 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:09.947191 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.947696 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.948341 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:09.948542 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:09.948628 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:09.950778 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:51199: BlockingRecv error: recv got EOF from 127.3.57.1:51199 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 07:00:09.951149 12212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:00:10.026582 12216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:10.026774 12216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:10.026798 12216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:10.027976 12216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:10.028012 12216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:10.028038 12216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:10.029382 12216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.2.keytab
--principal=oryx/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:37259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=45989
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:10.029556 12216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:10.029721 12216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:10.034680 12222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:10.034685 12221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:10.034694 12224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:10.034911 12216 server_base.cc:1034] running on GCE node
I20260927 07:00:10.035135 12216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:10.035391 12216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:10.036619 12216 hybrid_clock.cc:648] HybridClock initialized: now 1790492410036580 us; error 50 us; skew 500 ppm
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492410, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:10.039045 12216 init.cc:377] Logged in from keytab as oryx/127.3.57.2@KRBTEST.COM (short username oryx)
I20260927 07:00:10.039597 12216 webserver.cc:460] Webserver started at http://127.3.57.2:45989/ using document root <none> and password file <none>
I20260927 07:00:10.039836 12216 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:10.039915 12216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:10.041219 12216 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:10.041821 12231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.041983 12216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:10.042079 12216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "41ca6c60be4148e4ad094074a42751c7"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "79b8f5fa269759d8a9ba3271b595c0d0"
server_key_iv: "880d2e82969e3f06e8218b605cf1b9dd"
server_key_version: "encryptionkey@0"
I20260927 07:00:10.042421 12216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:10.053579 12216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:10.055385 12216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:10.055599 12216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:10.055831 12216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:10.056123 12216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:10.056177 12216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.056217 12216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:10.056244 12216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.061969 12216 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:37259
I20260927 07:00:10.062017 12344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:37259 every 8 connection(s)
I20260927 07:00:10.062358 12216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:10.064131 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12216
I20260927 07:00:10.064414 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:37995
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:37561
--webserver_port=37895
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:00:10.071205 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.063072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:37151 (local address 127.3.57.62:37561)
0927 07:00:10.063112 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.063114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.063331 (+ 217us) server_negotiation.cc:407] Connection header received
0927 07:00:10.063851 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.063852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.063863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.063923 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.064922 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.065668 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.067179 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.067524 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.067625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.071157 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:37151: BlockingRecv error: recv got EOF from 127.3.57.2:37151 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:00:10.071316 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.063050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:37151)
0927 07:00:10.063208 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.063248 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.063769 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.063984 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.063987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.064654 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.064830 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.064835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.065772 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.065773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.067084 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.067087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.067784 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.067786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.067922 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.068445 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.068453 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.071078 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
W20260927 07:00:10.071624 12345 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37561 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.57.62:37561: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:10.076419 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.071849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:34413)
0927 07:00:10.071897 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.071909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.071950 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.072120 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.072121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.072226 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.072266 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.072267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.073221 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.073223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.073730 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.073731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.074246 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.074247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.074305 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.074770 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.074774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.076377 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:00:10.076534 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.071931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:34413 (local address 127.3.57.62:37561)
0927 07:00:10.071998 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.072001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.072010 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:10.072035 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.072036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.072047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.072121 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.072335 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.073110 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.073818 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.074155 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.074233 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.076475 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:34413: BlockingRecv error: recv got EOF from 127.3.57.2:34413 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:10.080839 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.076910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:36375)
0927 07:00:10.076958 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.076967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.077026 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.077153 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.077154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.077268 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.077291 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.077292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.077916 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.077917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.078389 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.078391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.078745 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.078745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.078809 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.079182 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.079187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.080763 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:00:10.080943 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.076970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:36375 (local address 127.3.57.62:37561)
0927 07:00:10.077023 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.077025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.077034 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:10.077092 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.077093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.077101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.077141 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.077362 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.077807 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.078476 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.078663 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.078743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.080832 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:36375: BlockingRecv error: recv got EOF from 127.3.57.2:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 07:00:10.080951 12345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:00:10.146327 12348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:10.146534 12348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:10.146572 12348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:10.147791 12348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:10.147850 12348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:10.147897 12348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:10.149382 12348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/krb5kdc/oryx_127.3.57.3.keytab
--principal=oryx/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:37995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=37895
--tserver_master_addrs=127.3.57.62:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:10.149624 12348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:10.149849 12348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:10.154806 12354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:10.154796 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:10.154796 12353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:10.155098 12348 server_base.cc:1034] running on GCE node
I20260927 07:00:10.155277 12348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:10.155478 12348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:10.156615 12348 hybrid_clock.cc:648] HybridClock initialized: now 1790492410156603 us; error 29 us; skew 500 ppm
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492410, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:10.158933 12348 init.cc:377] Logged in from keytab as oryx/127.3.57.3@KRBTEST.COM (short username oryx)
I20260927 07:00:10.159328 12348 webserver.cc:460] Webserver started at http://127.3.57.3:37895/ using document root <none> and password file <none>
I20260927 07:00:10.159639 12348 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:10.159722 12348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:10.160995 12348 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:10.161550 12363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.161710 12348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:10.161804 12348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "ae2fa522335e40b2b6acb61b2331b396"
format_stamp: "Formatted at 2026-09-27 07:00:09 on dist-test-slave-mk3h"
server_key: "bd2281a6d2037c0d3e8e767555a6923e"
server_key_iv: "688b6dcffabaacac45c91cca01377817"
server_key_version: "encryptionkey@0"
I20260927 07:00:10.162150 12348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:10.171888 12348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:10.174062 12348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:10.174211 12348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:10.174422 12348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:10.174705 12348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:10.174759 12348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.174798 12348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:10.174825 12348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:10.180418 12348 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:37995
I20260927 07:00:10.180452 12476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:37995 every 8 connection(s)
I20260927 07:00:10.180786 12348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:00:10.187768 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.181554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:58183 (local address 127.3.57.62:37561)
0927 07:00:10.181629 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.181632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.181820 (+ 188us) server_negotiation.cc:407] Connection header received
0927 07:00:10.182284 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.182285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.182295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.182380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.183342 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.183911 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.184837 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.185020 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.185075 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.187723 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:58183: BlockingRecv error: recv got EOF from 127.3.57.3:58183 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:00:10.187778 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.181489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:58183)
0927 07:00:10.181695 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.181733 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.182171 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.182431 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.182435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.183056 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.183247 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.183251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.183985 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.183986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.184760 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.184762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.185090 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.185091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.185179 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.185668 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.185676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.187650 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
W20260927 07:00:10.187999 12477 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37561 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.57.62:37561: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
I20260927 07:00:10.189757 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12348
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:10.193102 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.188349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:39393)
0927 07:00:10.188411 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.188421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.188457 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.188683 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.188683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.188798 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.188841 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.188842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.189596 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.189597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.190318 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.190320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.190820 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.190821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.190869 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.191276 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.191281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.193058 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:00:10.193243 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.188526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:39393 (local address 127.3.57.62:37561)
0927 07:00:10.188582 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.188583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.188589 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:00:10.188606 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.188606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.188614 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.188679 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.188918 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.189488 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.190450 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.190743 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.190820 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.193171 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:39393: BlockingRecv error: recv got EOF from 127.3.57.3:39393 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:10.197806 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.190165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.3.57.62:37561)
0927 07:00:10.190413 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.190415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.190442 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:00:10.190465 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.190468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.190482 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.190534 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.191204 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.191892 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.192700 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.192891 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.192964 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.194358 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:10.194362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:10.194363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:10.194372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:10.196289 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:10.196687 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:10.196696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:10.196697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:10.196787 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:10.197012 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:10.197013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:10.197014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:10.197260 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:10.197407 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:10.197580 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 07:00:10.197711 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":181,"threads_started":1}
W20260927 07:00:10.198014 12215 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.193391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:41575 (local address 127.3.57.62:37561)
0927 07:00:10.193426 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.193428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.193525 (+ 97us) server_negotiation.cc:407] Connection header received
0927 07:00:10.193573 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.193574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.193584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.193645 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.193891 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.194521 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.195478 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.195706 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.195757 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.197969 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:41575: BlockingRecv error: recv got EOF from 127.3.57.3:41575 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:00:10.198199 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.193332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:41575)
0927 07:00:10.193449 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.193461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.193518 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.193655 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.193656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.193795 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.193819 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.193821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.194604 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.194606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.195401 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.195403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.195776 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.195777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.195829 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.196121 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.196126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.198142 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
W20260927 07:00:10.198320 12477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:10 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:10.955900 12483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:10.951529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:37117)
0927 07:00:10.951693 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:10.951704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:10.951782 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:10.951942 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:10.951943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:10.952086 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:10.952135 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.952136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.952765 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.952766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.953306 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:10.953308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.953619 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.953620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:10.953658 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.954079 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:10.954083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:10.955837 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":117,"threads_started":1}
W20260927 07:00:10.955996 12484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:10.951622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:37117 (local address 127.3.57.62:37561)
0927 07:00:10.951756 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:10.951758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:10.951768 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:10.951843 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:10.951844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:10.951852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:10.951916 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:10.952201 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.952701 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.953370 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:10.953560 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:10.953609 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:10.955924 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:37117: BlockingRecv error: recv got EOF from 127.3.57.1:37117 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260927 07:00:11.062996 12340 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.3.57.2:37259
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:11.085678 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:11.081297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:46193)
0927 07:00:11.081423 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:11.081433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:11.081488 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:11.081595 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:11.081595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:11.081740 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:11.081797 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.081799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.082404 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.082405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.082941 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.082942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.083437 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.083438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.083605 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.084000 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:11.084005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:11.085620 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260927 07:00:11.085762 12484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:11.081350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:46193 (local address 127.3.57.62:37561)
0927 07:00:11.081420 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:11.081422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:11.081492 (+ 70us) server_negotiation.cc:407] Connection header received
0927 07:00:11.081528 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:11.081529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:11.081539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:11.081586 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:11.081857 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.082343 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.083010 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.083354 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.083447 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.085702 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:46193: BlockingRecv error: recv got EOF from 127.3.57.2:46193 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:11.203478 12486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:11.198646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:47019)
0927 07:00:11.198807 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:11.198819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:11.198902 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:11.199056 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:11.199057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:11.199291 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:11.199336 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.199338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.200054 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.200055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.200591 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.200592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.200998 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.200999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.201053 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.201655 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:11.201660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:11.203405 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260927 07:00:11.203591 12484 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:11.198758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:47019 (local address 127.3.57.62:37561)
0927 07:00:11.198813 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:11.198815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:11.198884 (+ 69us) server_negotiation.cc:407] Connection header received
0927 07:00:11.198964 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:11.198965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:11.198978 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:11.199050 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:11.199410 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.199973 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.200694 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.200900 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.201000 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.203482 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:47019: BlockingRecv error: recv got EOF from 127.3.57.3:47019 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:11 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:11.960675 12487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:11.956407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:39223)
0927 07:00:11.956586 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:11.956596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:11.956650 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:11.956786 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:11.956786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:11.956933 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:11.956989 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.956991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.957644 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.957644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.958161 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:11.958163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.958477 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.958477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:11.958514 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.958937 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:11.958941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:11.960612 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
W20260927 07:00:11.960770 12488 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:11.956463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:39223 (local address 127.3.57.62:37561)
0927 07:00:11.956607 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:11.956609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:11.956656 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:00:11.956697 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:11.956697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:11.956707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:11.956769 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:11.957050 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.957576 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.958231 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:11.958415 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:11.958464 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:11.960699 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:39223: BlockingRecv error: recv got EOF from 127.3.57.1:39223 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:12.090521 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:12.086191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:47505)
0927 07:00:12.086330 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:12.086348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:12.086404 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:12.086515 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:12.086516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:12.086671 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:12.086719 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.086721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.087349 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.087350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.087890 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.087892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.088209 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.088210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.088247 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.088672 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:12.088677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:12.090446 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260927 07:00:12.090656 12488 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:12.086258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:47505 (local address 127.3.57.62:37561)
0927 07:00:12.086322 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:12.086324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:12.086408 (+ 84us) server_negotiation.cc:407] Connection header received
0927 07:00:12.086448 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:12.086449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:12.086460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:12.086510 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:12.086776 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.087270 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.087960 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.088145 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.088196 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.090588 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:47505: BlockingRecv error: recv got EOF from 127.3.57.2:47505 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:12.208567 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:12.203985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:38815)
0927 07:00:12.204137 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:12.204160 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:12.204210 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:12.204398 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:12.204400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:12.204577 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:12.204625 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.204627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.205295 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.205296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.205831 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.205832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.206187 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.206189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.206233 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.206770 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:12.206774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:12.208500 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
W20260927 07:00:12.208689 12488 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:12.204154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:38815 (local address 127.3.57.62:37561)
0927 07:00:12.204221 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:12.204224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:12.204261 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:00:12.204283 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:12.204284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:12.204297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:12.204344 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:12.204721 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.205217 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.205913 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.206111 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.206171 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.208594 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:38815: BlockingRecv error: recv got EOF from 127.3.57.3:38815 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:12 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:12.965884 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:12.961173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:52785)
0927 07:00:12.961338 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:12.961349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:12.961401 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:12.961551 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:12.961552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:12.961705 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:12.961752 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.961754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.962386 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.962387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.962933 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:12.962935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.963306 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.963306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:12.963344 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.963904 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:12.963909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:12.965818 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
W20260927 07:00:12.966010 12492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:12.961269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:52785 (local address 127.3.57.62:37561)
0927 07:00:12.961409 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:12.961411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:12.961421 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:12.961444 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:12.961444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:12.961454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:12.961546 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:12.961812 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.962322 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.963022 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:12.963232 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:12.963295 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:12.965902 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:52785: BlockingRecv error: recv got EOF from 127.3.57.1:52785 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:13.095933 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:13.091123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:58327)
0927 07:00:13.091252 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:13.091262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:13.091318 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:13.091451 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:13.091452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:13.091677 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:13.091727 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.091729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.092361 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.092362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.092869 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.092870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.093352 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.093354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.093417 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.093929 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:13.093934 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:13.095825 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260927 07:00:13.095965 12492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:13.091193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:58327 (local address 127.3.57.62:37561)
0927 07:00:13.091261 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:13.091263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:13.091322 (+ 59us) server_negotiation.cc:407] Connection header received
0927 07:00:13.091364 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:13.091364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:13.091376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:13.091430 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:13.091786 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.092296 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.092958 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.093257 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.093324 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.095901 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:58327: BlockingRecv error: recv got EOF from 127.3.57.2:58327 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:13.213786 12494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:13.209080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:42397)
0927 07:00:13.209255 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:13.209269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:13.209346 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:13.209476 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:13.209476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:13.209650 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:13.209695 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.209697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.210367 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.210368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.210917 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.210919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.211315 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.211316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.211378 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.211843 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:13.211847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:13.213719 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
W20260927 07:00:13.213904 12492 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:13.209188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:42397 (local address 127.3.57.62:37561)
0927 07:00:13.209273 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:13.209276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:13.209341 (+ 65us) server_negotiation.cc:407] Connection header received
0927 07:00:13.209403 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:13.209404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:13.209420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:13.209471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:13.209776 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.210284 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.211004 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.211218 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.211314 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.213807 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:42397: BlockingRecv error: recv got EOF from 127.3.57.3:42397 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:13 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:13.970808 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:13.966297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:40587)
0927 07:00:13.966423 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:13.966434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:13.966480 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:13.966663 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:13.966665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:13.966822 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:13.966872 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.966873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.967492 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.967493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.968078 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:13.968080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.968420 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.968421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:13.968473 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.968966 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:13.968971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:13.970748 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
W20260927 07:00:13.970937 12496 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:13.966404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:40587 (local address 127.3.57.62:37561)
0927 07:00:13.966541 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:13.966542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:13.966549 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:00:13.966570 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:13.966570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:13.966590 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:13.966650 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:13.966938 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.967430 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.968153 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:13.968338 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:13.968389 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:13.970843 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:40587: BlockingRecv error: recv got EOF from 127.3.57.1:40587 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:14.105186 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:14.096485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:45625)
0927 07:00:14.096628 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:14.096641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:14.096700 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:14.096838 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:14.096839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:14.096991 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:14.097037 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.097039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.097742 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.097744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.098790 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.098792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.099173 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.099174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.099227 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.103323 (+ 4096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:14.103336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:14.105129 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
W20260927 07:00:14.105283 12496 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:14.096531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:45625 (local address 127.3.57.62:37561)
0927 07:00:14.096640 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:14.096644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:14.096682 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:00:14.096765 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:14.096765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:14.096776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:14.096825 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:14.097094 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.097657 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.098863 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.099089 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.099143 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.105216 (+ 6073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:45625: BlockingRecv error: recv got EOF from 127.3.57.2:45625 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:14.218690 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:14.214289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:47047)
0927 07:00:14.214404 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:14.214417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:14.214489 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:14.214661 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:14.214662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:14.214846 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:14.214894 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.214895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.215516 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.215516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.216153 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.216155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.216490 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.216490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.216526 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.216955 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:14.216959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:14.218635 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
W20260927 07:00:14.218820 12496 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:14.214358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:47047 (local address 127.3.57.62:37561)
0927 07:00:14.214407 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:14.214410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:14.214491 (+ 81us) server_negotiation.cc:407] Connection header received
0927 07:00:14.214540 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:14.214541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:14.214556 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:14.214636 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:14.214965 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.215455 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.216225 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.216409 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.216499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.218739 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:47047: BlockingRecv error: recv got EOF from 127.3.57.3:47047 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:14 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:14.975816 12499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:14.971300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:45743)
0927 07:00:14.971452 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:14.971463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:14.971516 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:14.971761 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:14.971762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:14.971923 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:14.971980 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.971982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.972613 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.972614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.973146 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:14.973148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.973462 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.973462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:14.973503 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.973927 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:14.973932 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:14.975757 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
W20260927 07:00:14.975946 12500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:14.971388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:45743 (local address 127.3.57.62:37561)
0927 07:00:14.971508 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:14.971510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:14.971518 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:14.971639 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:14.971640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:14.971650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:14.971722 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:14.972056 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.972542 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.973217 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:14.973398 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:14.973445 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:14.975837 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:45743: BlockingRecv error: recv got EOF from 127.3.57.1:45743 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:15.110070 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:15.105761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:55009)
0927 07:00:15.105948 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:15.105961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:15.106029 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:15.106173 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:15.106174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:15.106330 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:15.106379 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.106381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.106979 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.106979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.107496 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.107497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.107866 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.107867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.107908 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.108321 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:15.108325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:15.110014 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
W20260927 07:00:15.110204 12500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:15.105874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:55009 (local address 127.3.57.62:37561)
0927 07:00:15.105919 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:15.105921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:15.106032 (+ 111us) server_negotiation.cc:407] Connection header received
0927 07:00:15.106100 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:15.106100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:15.106110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:15.106152 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:15.106441 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.106919 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.107585 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.107799 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.107855 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.110128 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:55009: BlockingRecv error: recv got EOF from 127.3.57.2:55009 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:15.223259 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:15.219099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:43561)
0927 07:00:15.219229 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:15.219265 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:15.219321 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:15.219439 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:15.219439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:15.219616 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:15.219661 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.219663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.220276 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.220277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.220794 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.220795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.221099 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.221100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.221135 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.221555 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:15.221559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:15.223202 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
W20260927 07:00:15.223351 12500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:15.219154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:43561 (local address 127.3.57.62:37561)
0927 07:00:15.219221 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:15.219228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:15.219325 (+ 97us) server_negotiation.cc:407] Connection header received
0927 07:00:15.219373 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:15.219374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:15.219386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:15.219434 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:15.219722 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.220207 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.220862 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.221039 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.221087 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.223284 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:43561: BlockingRecv error: recv got EOF from 127.3.57.3:43561 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:15 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:15.980580 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:15.976230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:36575)
0927 07:00:15.976375 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:15.976386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:15.976438 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:15.976562 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:15.976563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:15.976714 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:15.976760 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.976762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.977360 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.977361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.977854 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:15.977856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.978151 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.978152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:15.978199 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.978741 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:15.978747 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:15.980485 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
W20260927 07:00:15.980634 12504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:15.976319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:36575 (local address 127.3.57.62:37561)
0927 07:00:15.976436 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:15.976438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:15.976447 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:15.976481 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:15.976482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:15.976491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:15.976546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:15.976819 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.977298 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.977918 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:15.978096 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:15.978141 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:15.980559 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:36575: BlockingRecv error: recv got EOF from 127.3.57.1:36575 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:16.114696 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:16.110545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:49283)
0927 07:00:16.110677 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:16.110688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:16.110750 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:16.110867 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:16.110868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:16.111010 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:16.111056 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.111057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.111726 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.111727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.112228 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.112229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.112535 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.112536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.112572 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.113021 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:16.113025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:16.114633 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
W20260927 07:00:16.114806 12504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:16.110597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:49283 (local address 127.3.57.62:37561)
0927 07:00:16.110675 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:16.110677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:16.110759 (+ 82us) server_negotiation.cc:407] Connection header received
0927 07:00:16.110789 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:16.110789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:16.110800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:16.110853 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:16.111116 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.111651 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.112295 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.112473 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.112523 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.114731 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:49283: BlockingRecv error: recv got EOF from 127.3.57.2:49283 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:16.228643 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:16.223766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:44179)
0927 07:00:16.223972 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:16.223982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:16.224057 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:16.224216 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:16.224217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:16.224438 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:16.224494 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.224496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.225234 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.225235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.225845 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.225847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.226266 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.226268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.226337 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.226865 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:16.226871 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:16.228553 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
W20260927 07:00:16.228701 12504 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:16.223834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:44179 (local address 127.3.57.62:37561)
0927 07:00:16.223904 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:16.223906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:16.224037 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:00:16.224120 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:16.224121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:16.224134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:16.224175 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:16.224568 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.225151 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.225938 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.226198 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.226269 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.228629 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:44179: BlockingRecv error: recv got EOF from 127.3.57.3:44179 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:16 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:16.985273 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:16.981043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:52103)
0927 07:00:16.981165 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:16.981175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:16.981225 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:16.981350 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:16.981351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:16.981517 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:16.981566 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.981567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.982189 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.982189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.982734 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:16.982736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.983049 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.983049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:16.983094 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.983511 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:16.983515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:16.985215 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260927 07:00:16.985338 12508 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:16.981091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:52103 (local address 127.3.57.62:37561)
0927 07:00:16.981221 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:16.981223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:16.981232 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:00:16.981266 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:16.981266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:16.981276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:16.981334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:16.981628 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.982124 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.982803 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:16.982983 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:16.983030 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:16.985288 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:52103: BlockingRecv error: recv got EOF from 127.3.57.1:52103 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:17.120256 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:17.115215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:47817)
0927 07:00:17.115485 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:17.115498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:17.115591 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:17.115790 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:17.115791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:17.115976 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:17.116021 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.116023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.116703 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.116705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.117383 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.117386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.117758 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.117759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.117815 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.118311 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:17.118316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:17.120187 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":167,"threads_started":1}
W20260927 07:00:17.120373 12508 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:17.115406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:47817 (local address 127.3.57.62:37561)
0927 07:00:17.115498 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:17.115501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:17.115566 (+ 65us) server_negotiation.cc:407] Connection header received
0927 07:00:17.115686 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:17.115687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:17.115714 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:17.115784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:17.116102 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.116616 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.117476 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.117668 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.117757 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.120272 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:47817: BlockingRecv error: recv got EOF from 127.3.57.2:47817 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:17.233713 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:17.229100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:40363)
0927 07:00:17.229289 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:17.229301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:17.229365 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:17.229571 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:17.229572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:17.229755 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:17.229802 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.229805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.230429 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.230430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.230961 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.230962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.231262 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.231263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.231299 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.231758 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:17.231762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:17.233600 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
W20260927 07:00:17.233743 12508 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:17.229135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:40363 (local address 127.3.57.62:37561)
0927 07:00:17.229201 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:17.229204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:17.229366 (+ 162us) server_negotiation.cc:407] Connection header received
0927 07:00:17.229453 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:17.229454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:17.229484 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:17.229571 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:17.229867 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.230362 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.231026 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.231201 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.231250 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.233697 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:40363: BlockingRecv error: recv got EOF from 127.3.57.3:40363 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:17 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:17.990383 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:17.985768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:47395)
0927 07:00:17.985911 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:17.985923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:17.985997 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:17.986136 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:17.986137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:17.986303 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:17.986350 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.986352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.987126 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.987127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.987710 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:17.987712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.988069 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.988070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:17.988137 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.988640 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:17.988645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:17.990277 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
W20260927 07:00:17.990437 12512 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:17.985866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:47395 (local address 127.3.57.62:37561)
0927 07:00:17.986000 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:17.986002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:17.986010 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:17.986063 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:17.986064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:17.986075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:17.986133 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:17.986435 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.987030 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.987798 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:17.987990 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:17.988058 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:17.990367 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:47395: BlockingRecv error: recv got EOF from 127.3.57.1:47395 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:18.125535 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:18.120677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:52739)
0927 07:00:18.120913 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:18.120924 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:18.120979 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:18.121116 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:18.121117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:18.121283 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:18.121331 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.121333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.122167 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.122168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.122722 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.122723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.123050 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.123051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.123090 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.123604 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:18.123609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:18.125477 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":179,"threads_started":1}
W20260927 07:00:18.125640 12512 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:18.120777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:52739 (local address 127.3.57.62:37561)
0927 07:00:18.120846 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:18.120848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:18.120990 (+ 142us) server_negotiation.cc:407] Connection header received
0927 07:00:18.121032 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:18.121033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:18.121043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:18.121087 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:18.121430 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.122085 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.122797 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.122984 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.123049 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.125567 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:52739: BlockingRecv error: recv got EOF from 127.3.57.2:52739 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:18.239148 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:18.234301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:34801)
0927 07:00:18.234542 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:18.234554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:18.234628 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:18.234776 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:18.234777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:18.234958 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:18.235004 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.235005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.235735 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.235736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.236261 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.236262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.236615 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.236616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.236655 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.237208 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:18.237214 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:18.239048 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":151,"threads_started":1}
W20260927 07:00:18.239198 12512 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:18.234381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:34801 (local address 127.3.57.62:37561)
0927 07:00:18.234457 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:18.234460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:18.234651 (+ 191us) server_negotiation.cc:407] Connection header received
0927 07:00:18.234682 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:18.234683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:18.234697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:18.234755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:18.235063 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.235659 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.236349 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.236543 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.236605 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.239135 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:34801: BlockingRecv error: recv got EOF from 127.3.57.3:34801 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:18 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:18.995216 12515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:18.990875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:53841)
0927 07:00:18.991021 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:18.991032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:18.991079 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:18.991218 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:18.991219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:18.991372 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:18.991418 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.991419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.992103 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.992104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.992613 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:18.992615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.992926 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.992927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:18.992963 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.993466 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:18.993471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:18.995159 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260927 07:00:18.995337 12516 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:18.990951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:53841 (local address 127.3.57.62:37561)
0927 07:00:18.991091 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:18.991093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:18.991108 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:00:18.991128 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:18.991129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:18.991139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:18.991197 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:18.991510 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.992028 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.992683 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:18.992867 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:18.992917 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:18.995251 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:53841: BlockingRecv error: recv got EOF from 127.3.57.1:53841 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:19.144951 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:19.125965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:37881)
0927 07:00:19.126113 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:19.126126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:19.126213 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:19.126383 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:19.126384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:19.126581 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:19.126626 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.126627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.127288 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.127289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.127866 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.127868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.128232 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.128232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.128294 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:19.143069 (+ 14775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:19.143084 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:19.144847 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260927 07:00:19.145023 12516 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:19.126040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:37881 (local address 127.3.57.62:37561)
0927 07:00:19.126118 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:19.126120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:19.126193 (+ 73us) server_negotiation.cc:407] Connection header received
0927 07:00:19.126286 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:19.126287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:19.126300 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:19.126373 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:19.126714 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.127206 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.127951 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.128148 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.128232 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:19.144924 (+ 16692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:37881: BlockingRecv error: recv got EOF from 127.3.57.2:37881 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:19.244613 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:19.239634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:34651)
0927 07:00:19.239822 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:19.239835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:19.239951 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:19.240163 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:19.240165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:19.240348 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:19.240396 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.240398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.241124 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.241125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.241740 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.241741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.242176 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.242177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.242238 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:19.242760 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:19.242768 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:19.244539 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":107,"threads_started":1}
W20260927 07:00:19.244896 12516 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:19.239708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:34651 (local address 127.3.57.62:37561)
0927 07:00:19.239808 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:19.239811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:19.239931 (+ 120us) server_negotiation.cc:407] Connection header received
0927 07:00:19.240039 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:19.240040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:19.240057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:19.240134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:19.240481 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.241033 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.241833 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.242075 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.242174 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:19.244815 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:34651: BlockingRecv error: recv got EOF from 127.3.57.3:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:19 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:20.000723 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:19.995737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:42515)
0927 07:00:19.995895 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:19.995906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:19.995977 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:19.996144 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:19.996146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:19.996304 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:19.996353 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.996354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.997048 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.997049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.997663 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:19.997665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.998102 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.998103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:19.998199 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:19.998735 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:19.998740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:20.000627 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
W20260927 07:00:20.000780 12520 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:19.995819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:42515 (local address 127.3.57.62:37561)
0927 07:00:19.995925 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:19.995927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:19.995959 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:00:19.996029 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:19.996030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:19.996042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:19.996103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:19.996462 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.996960 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.997749 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:19.998047 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:19.998121 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:20.000709 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:42515: BlockingRecv error: recv got EOF from 127.3.57.1:42515 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
Sep 27 07:00:20 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:20 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:20.149945 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:20.145450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:46545)
0927 07:00:20.145605 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:20.145615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:20.145678 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:20.145798 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:20.145799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:20.145970 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:20.146015 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:20.146016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.146621 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.146621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:20.147164 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:20.147166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.147485 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.147486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:20.147534 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:20.147960 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:20.147965 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:20.149890 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
W20260927 07:00:20.150069 12520 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:20.145564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:46545 (local address 127.3.57.62:37561)
0927 07:00:20.145615 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:20.145621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:20.145662 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:00:20.145731 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:20.145732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:20.145742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:20.145783 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:20.146062 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.146557 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.147237 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.147421 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.147474 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:20.149989 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:46545: BlockingRecv error: recv got EOF from 127.3.57.2:46545 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 07:00:20 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:20 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:20.249666 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:20.245131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:53517)
0927 07:00:20.245278 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:20.245292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:20.245375 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:20.245563 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:20.245564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:20.245773 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:20.245837 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:20.245840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.246530 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.246530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:20.247056 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:20.247057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.247381 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.247382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:20.247419 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:20.247933 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:20.247938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:20.249612 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
W20260927 07:00:20.249790 12520 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:20.245266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:53517 (local address 127.3.57.62:37561)
0927 07:00:20.245313 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:20.245315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:20.245345 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:00:20.245423 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:20.245424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:20.245441 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:20.245557 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:20.245910 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.246467 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.247127 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:20.247322 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:20.247369 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:20.249693 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:53517: BlockingRecv error: recv got EOF from 127.3.57.3:53517 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:21.006021 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:21.001204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:57269)
0927 07:00:21.001400 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:21.001412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:21.001483 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:21.001657 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:21.001658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:21.001829 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:21.001877 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.001879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.002555 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.002556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.003085 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.003087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.003431 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.003432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.003498 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.004160 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:21.004166 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:21.005962 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
W20260927 07:00:21.006167 12524 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:21.001293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:57269 (local address 127.3.57.62:37561)
0927 07:00:21.001402 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:21.001404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:21.001486 (+ 82us) server_negotiation.cc:407] Connection header received
0927 07:00:21.001527 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:21.001527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:21.001538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:21.001598 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:21.001948 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.002452 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.003159 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.003341 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.003389 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.006047 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:57269: BlockingRecv error: recv got EOF from 127.3.57.1:57269 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:21.154807 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:21.150495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:59317)
0927 07:00:21.150626 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:21.150635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:21.150691 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:21.150869 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:21.150870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:21.151031 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:21.151076 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.151077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.151724 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.151725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.152247 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.152249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.152606 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.152606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.152655 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.153089 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:21.153093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:21.154749 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
W20260927 07:00:21.154907 12524 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:21.150578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:59317 (local address 127.3.57.62:37561)
0927 07:00:21.150662 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:21.150664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:21.150677 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:00:21.150782 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:21.150783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:21.150795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:21.150852 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:21.151133 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.151656 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.152339 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.152534 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.152598 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.154841 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:59317: BlockingRecv error: recv got EOF from 127.3.57.2:59317 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:21 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:21.254660 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:21.250079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:60361)
0927 07:00:21.250266 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:21.250284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:21.250353 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:21.250577 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:21.250578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:21.250757 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:21.250804 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.250806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.251485 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.251486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.252033 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:21.252035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.252415 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.252415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:21.252477 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.252913 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:21.252917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:21.254569 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
W20260927 07:00:21.254722 12524 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:21.250210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:60361 (local address 127.3.57.62:37561)
0927 07:00:21.250315 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:21.250317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:21.250329 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:00:21.250435 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:21.250436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:21.250451 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:21.250565 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:21.250890 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.251395 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.252125 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:21.252326 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:21.252414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:21.254658 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:60361: BlockingRecv error: recv got EOF from 127.3.57.3:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:22.011003 12527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:22.006557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:57479)
0927 07:00:22.006714 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:22.006726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:22.006782 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:22.006909 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:22.006910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:22.007062 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:22.007116 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.007118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.007791 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.007792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.008321 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.008322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.008634 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.008634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.008673 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.009167 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:22.009172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:22.010935 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
W20260927 07:00:22.011159 12528 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:22.006660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:57479 (local address 127.3.57.62:37561)
0927 07:00:22.006794 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:22.006795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:22.006802 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:00:22.006823 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:22.006824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:22.006834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:22.006894 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:22.007181 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.007724 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.008390 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.008572 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.008632 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.011062 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:57479: BlockingRecv error: recv got EOF from 127.3.57.1:57479 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:22.159632 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:22.155249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:59693)
0927 07:00:22.155361 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:22.155373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:22.155435 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:22.155612 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:22.155612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:22.155769 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:22.155817 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.155819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.156433 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.156434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.156939 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.156940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.157252 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.157253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.157303 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.157805 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:22.157817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:22.159502 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
W20260927 07:00:22.159667 12528 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:22.155319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:59693 (local address 127.3.57.62:37561)
0927 07:00:22.155405 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:22.155407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:22.155419 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:00:22.155494 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:22.155495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:22.155505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:22.155606 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:22.155878 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.156367 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.157007 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.157190 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.157239 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.159609 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:59693: BlockingRecv error: recv got EOF from 127.3.57.2:59693 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:22 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:22.259683 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:22.255158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:33421)
0927 07:00:22.255304 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:22.255314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:22.255374 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:22.255505 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:22.255506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:22.255726 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:22.255774 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.255776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.256391 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.256392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.256881 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:22.256883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.257199 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.257200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:22.257235 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.257757 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:22.257762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:22.259542 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
W20260927 07:00:22.259696 12528 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:22.255218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:33421 (local address 127.3.57.62:37561)
0927 07:00:22.255308 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:22.255310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:22.255380 (+ 70us) server_negotiation.cc:407] Connection header received
0927 07:00:22.255418 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:22.255418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:22.255430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:22.255478 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:22.255847 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.256326 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.256954 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:22.257136 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:22.257187 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:22.259637 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:33421: BlockingRecv error: recv got EOF from 127.3.57.3:33421 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:23.016161 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:23.011448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:38551)
0927 07:00:23.011639 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:23.011651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:23.011714 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:23.011848 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:23.011848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:23.012004 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:23.012052 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.012054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.012680 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.012681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.013197 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.013200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.013517 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.013518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.013555 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.013969 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:23.013974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:23.016055 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":131,"threads_started":1}
W20260927 07:00:23.016209 12532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:23.011566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:38551 (local address 127.3.57.62:37561)
0927 07:00:23.011685 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:23.011687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:23.011705 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:00:23.011765 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:23.011765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:23.011776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:23.011833 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:23.012116 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.012614 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.013267 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.013456 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.013507 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.016131 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:38551: BlockingRecv error: recv got EOF from 127.3.57.1:38551 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:23.164728 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:23.160107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:57667)
0927 07:00:23.160270 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:23.160281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:23.160347 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:23.160475 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:23.160475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:23.160678 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:23.160727 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.160729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.161372 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.161373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.161916 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.161917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.162235 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.162235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.162273 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.162716 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:23.162721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:23.164622 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
W20260927 07:00:23.164849 12532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:23.160227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:57667 (local address 127.3.57.62:37561)
0927 07:00:23.160289 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:23.160291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:23.160340 (+ 49us) server_negotiation.cc:407] Connection header received
0927 07:00:23.160393 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:23.160394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:23.160404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:23.160453 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:23.160793 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.161286 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.161984 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.162174 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.162222 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.164763 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:57667: BlockingRecv error: recv got EOF from 127.3.57.2:57667 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:23 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:23.264909 12534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:23.260136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:47529)
0927 07:00:23.260313 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:23.260324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:23.260389 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:23.260545 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:23.260546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:23.260697 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:23.260743 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.260744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.261354 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.261355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.262127 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:23.262134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.262466 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.262467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:23.262530 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.262997 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:23.263001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:23.264837 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
W20260927 07:00:23.264973 12532 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:23.260219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:47529 (local address 127.3.57.62:37561)
0927 07:00:23.260293 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:23.260300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:23.260387 (+ 87us) server_negotiation.cc:407] Connection header received
0927 07:00:23.260440 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:23.260440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:23.260452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:23.260493 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:23.260801 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.261293 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.262196 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:23.262403 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:23.262451 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:23.264924 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:47529: BlockingRecv error: recv got EOF from 127.3.57.3:47529 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:24.021274 12535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:24.016685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:54503)
0927 07:00:24.016859 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:24.016870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:24.016931 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:24.017070 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:24.017071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:24.017228 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:24.017276 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.017277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.018006 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.018007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.018526 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.018528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.018885 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.018886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.018924 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.019357 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:24.019361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:24.021206 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
W20260927 07:00:24.021399 12536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:24.016775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:54503 (local address 127.3.57.62:37561)
0927 07:00:24.016891 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:24.016893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:24.016915 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:00:24.016973 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:24.016974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:24.016984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:24.017039 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:24.017344 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.017922 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.018618 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.018818 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.018870 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.021293 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:54503: BlockingRecv error: recv got EOF from 127.3.57.1:54503 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:24.174876 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:24.165364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:53317)
0927 07:00:24.165536 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:24.165546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:24.165648 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:24.165802 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:24.165803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:24.165979 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:24.166025 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.166027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.166760 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.166761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.167295 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.167296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.167727 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.167727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.167781 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.173113 (+ 5332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:24.173118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:24.174818 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
W20260927 07:00:24.175024 12536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:24.165445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:53317 (local address 127.3.57.62:37561)
0927 07:00:24.165517 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:24.165519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:24.165628 (+ 109us) server_negotiation.cc:407] Connection header received
0927 07:00:24.165733 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:24.165734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:24.165747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:24.165797 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:24.166141 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.166667 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.167381 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.167645 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.167727 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.174922 (+ 7195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:53317: BlockingRecv error: recv got EOF from 127.3.57.2:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:24 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.3@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:24.270134 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:24.265355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.3:36325)
0927 07:00:24.265639 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:24.265651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:24.265739 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:24.265890 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:24.265891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:24.266071 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:24.266128 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.266130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.266833 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.266834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.267384 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:24.267386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.267801 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.267802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:24.267863 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.268372 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:24.268377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:24.270057 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
W20260927 07:00:24.270203 12536 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:24.265433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:36325 (local address 127.3.57.62:37561)
0927 07:00:24.265608 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:24.265610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:24.265719 (+ 109us) server_negotiation.cc:407] Connection header received
0927 07:00:24.265802 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:24.265803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:24.265824 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:24.265884 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:24.266210 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.266741 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.267469 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:24.267717 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:24.267799 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:24.270138 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.3:36325: BlockingRecv error: recv got EOF from 127.3.57.3:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 27 07:00:25 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:25 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.1@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:25.026364 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.021795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.1:37455)
0927 07:00:25.021987 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:25.021998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:25.022061 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:25.022191 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:25.022192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:25.022351 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:25.022397 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.022399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.023183 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.023184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.023734 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.023736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.024048 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.024048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.024084 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.024553 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:25.024557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:25.026288 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
W20260927 07:00:25.026525 12540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:25.021888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:37455 (local address 127.3.57.62:37561)
0927 07:00:25.021995 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.021997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.022045 (+ 48us) server_negotiation.cc:407] Connection header received
0927 07:00:25.022112 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.022113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.022123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.022170 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.022482 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.023115 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.023798 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.023985 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.024048 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.026414 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.1:37455: BlockingRecv error: recv got EOF from 127.3.57.1:37455 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
Sep 27 07:00:25 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:00:25 dist-test-slave-mk3h krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.57.2@KRBTEST.COM for oryx/127.3.57.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:00:25.181437 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.175956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37561 (local address 127.3.57.2:37055)
0927 07:00:25.176092 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:25.176104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:25.176167 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:25.176639 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:25.176640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:25.176816 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:25.176867 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.176869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.177588 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.177590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.178374 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.178377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.178762 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.178763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.178809 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.179273 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:25.179278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:25.181341 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.57.62:37561: Server oryx/127.3.57.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260927 07:00:25.181486 12540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:25.176346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:37055 (local address 127.3.57.62:37561)
0927 07:00:25.176394 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.176396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.176412 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:00:25.176442 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.176442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.176452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.176516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.177001 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.177502 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.178485 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.178682 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.178750 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.181420 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.57.2:37055: BlockingRecv error: recv got EOF from 127.3.57.2:37055 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:25.191339 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12064
I20260927 07:00:25.195657 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12216
I20260927 07:00:25.199838 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12348
I20260927 07:00:25.203998 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 11994
2026-09-27T07:00:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17389 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T07:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:25Z Disabled control of system clock
I20260927 07:00:25.231423 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34909
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:34909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 07:00:25.308872 12545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:25.309064 12545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:25.309084 12545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:25.310177 12545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:25.310211 12545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:25.310223 12545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:25.310236 12545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:25.310248 12545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:25.311513 12545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:34909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:25.311707 12545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:25.311931 12545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:25.317407 12553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.317358 12545 server_base.cc:1034] running on GCE node
W20260927 07:00:25.317360 12550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.317368 12551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.317812 12545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:25.318101 12545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:25.319242 12545 hybrid_clock.cc:648] HybridClock initialized: now 1790492425319223 us; error 37 us; skew 500 ppm
I20260927 07:00:25.320632 12545 webserver.cc:460] Webserver started at http://127.3.57.62:37617/ using document root <none> and password file <none>
I20260927 07:00:25.320886 12545 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:25.320933 12545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:25.321102 12545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:25.322149 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "427c7e1e79d14d5e8bd495931860db5c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "8ccc428a92acc7a70735d7aba6579248"
server_key_iv: "9f690c255996d00368c08b21d95049c5"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.322502 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "427c7e1e79d14d5e8bd495931860db5c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "8ccc428a92acc7a70735d7aba6579248"
server_key_iv: "9f690c255996d00368c08b21d95049c5"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.323942 12545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:00:25.324709 12559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.324894 12545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:25.324982 12545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "427c7e1e79d14d5e8bd495931860db5c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "8ccc428a92acc7a70735d7aba6579248"
server_key_iv: "9f690c255996d00368c08b21d95049c5"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.325049 12545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:25.359829 12545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:25.360100 12545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:25.360200 12545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:25.363996 12545 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:34909
I20260927 07:00:25.364030 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:34909 every 8 connection(s)
I20260927 07:00:25.364377 12545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:25.365060 12612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:25.365705 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12545
I20260927 07:00:25.365777 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:00:25.366004 3300 external_mini_cluster.cc:1428] Setting key a6e668a0b886ed8d2d1ffd818c7db862
I20260927 07:00:25.368355 12612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c: Bootstrap starting.
I20260927 07:00:25.368969 12612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:25.369306 12612 log.cc:824] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:25.370057 12612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c: No bootstrap required, opened a new log
I20260927 07:00:25.372205 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.366808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.3.57.62:34909)
0927 07:00:25.367032 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.367040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.367067 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:00:25.367696 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.367703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.367769 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.367917 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.368160 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.369078 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.370207 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.370652 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.370753 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.371444 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:25.371447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:25.371448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:25.371679 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:25.371713 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:25.371725 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:25.371776 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:25.371906 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 07:00:25.372010 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260927 07:00:25.372346 12612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } }
I20260927 07:00:25.372524 12612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:25.372571 12612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427c7e1e79d14d5e8bd495931860db5c, State: Initialized, Role: FOLLOWER
I20260927 07:00:25.372696 12612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } }
I20260927 07:00:25.372772 12612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:25.372816 12612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:25.372866 12612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:25.373397 12612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } }
I20260927 07:00:25.373502 12612 leader_election.cc:302] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427c7e1e79d14d5e8bd495931860db5c; no voters:
I20260927 07:00:25.373646 12612 leader_election.cc:288] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:00:25.373766 12617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:25.373919 12612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:25.373922 12617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [term 1 LEADER]: Becoming Leader. State: Replica: 427c7e1e79d14d5e8bd495931860db5c, State: Running, Role: LEADER
I20260927 07:00:25.374148 12617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } }
I20260927 07:00:25.374517 12618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427c7e1e79d14d5e8bd495931860db5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } } }
I20260927 07:00:25.374533 12619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427c7e1e79d14d5e8bd495931860db5c. Latest consensus state: current_term: 1 leader_uuid: "427c7e1e79d14d5e8bd495931860db5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427c7e1e79d14d5e8bd495931860db5c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34909 } } }
I20260927 07:00:25.374624 12618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:25.374627 12619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:25.374812 12626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:25.375451 12626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:25.376889 12626 catalog_manager.cc:1347] Generated new cluster ID: ba189536732949a5b07db5db9370542d
I20260927 07:00:25.376941 12626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:00:25.400574 12626 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:25.401124 12626 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:25.403800 12626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427c7e1e79d14d5e8bd495931860db5c: Generated new TSK 0
I20260927 07:00:25.403996 12626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:00:25.408910 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--builtin_ntp_servers=127.3.57.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 07:00:25.488776 12636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:25.488970 12636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:25.488991 12636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:25.490130 12636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:25.490170 12636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:25.490193 12636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:25.491511 12636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:25.491775 12636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:25.492069 12636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:25.497692 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.497709 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.497888 12636 server_base.cc:1034] running on GCE node
W20260927 07:00:25.497828 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.498111 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:25.498373 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:25.499532 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790492425499512 us; error 59 us; skew 500 ppm
I20260927 07:00:25.505785 12636 webserver.cc:460] Webserver started at http://127.3.57.1:42627/ using document root <none> and password file <none>
I20260927 07:00:25.506018 12636 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:25.506089 12636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:25.506242 12636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:25.507277 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "644a5127d5ac47138cc87ee51265db9b"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "35c5d5d43ccc4fe94221bfbb408e9169"
server_key_iv: "70c2f3a51c1aa9600edf92279ed60d72"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.507694 12636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "644a5127d5ac47138cc87ee51265db9b"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "35c5d5d43ccc4fe94221bfbb408e9169"
server_key_iv: "70c2f3a51c1aa9600edf92279ed60d72"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.508946 12636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:00:25.509660 12650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.509817 12636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:25.509913 12636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "644a5127d5ac47138cc87ee51265db9b"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "35c5d5d43ccc4fe94221bfbb408e9169"
server_key_iv: "70c2f3a51c1aa9600edf92279ed60d72"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.509999 12636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:25.524998 12636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:25.525281 12636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:25.525431 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:25.525645 12636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:25.525954 12636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:25.526006 12636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.526038 12636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:25.526065 12636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.532063 12636 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:43857
I20260927 07:00:25.532105 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:43857 every 8 connection(s)
I20260927 07:00:25.532418 12636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:00:25.533603 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12636
I20260927 07:00:25.533690 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:00:25.533905 3300 external_mini_cluster.cc:1428] Setting key 1feffffe16e665c3680b95916aa4bb43
I20260927 07:00:25.534739 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--builtin_ntp_servers=127.3.57.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 07:00:25.537642 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.533067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:55953 (local address 127.3.57.62:34909)
0927 07:00:25.533125 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.533128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.533359 (+ 231us) server_negotiation.cc:407] Connection header received
0927 07:00:25.533942 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.533943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.533956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.534010 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.534670 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.535259 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.536279 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.536480 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.536560 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.537347 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:25.537349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:25.537349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:25.537353 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:25.537367 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:25.537371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:25.537437 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:25.537533 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:00:25.537562 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:00:25.537905 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.533064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34909 (local address 127.3.57.1:55953)
0927 07:00:25.533248 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:25.533279 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:25.533843 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:25.534093 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:25.534096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:25.534104 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:25.534579 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.534583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.535356 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.535357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.536181 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.536183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.536565 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.536567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.536693 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.537199 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:25.537199 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:25.537226 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:25.537252 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:25.537439 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:25.537445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:25.537447 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:00:25.537557 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:00:25.537654 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260927 07:00:25.538335 12764 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34909
I20260927 07:00:25.538432 12764 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:25.538625 12764 heartbeater.cc:511] Master 127.3.57.62:34909 requested a full tablet report, sending...
I20260927 07:00:25.539129 12576 ts_manager.cc:194] Registered new tserver with Master: 644a5127d5ac47138cc87ee51265db9b (127.3.57.1:43857)
I20260927 07:00:25.539786 12576 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.1:55953
W20260927 07:00:25.623495 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:25.623804 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:25.623843 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:25.624989 12767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:25.625051 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:25.625094 12767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:25.626417 12767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:25.626694 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:25.626986 12767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:25.631865 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.631922 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.631976 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.632184 12767 server_base.cc:1034] running on GCE node
I20260927 07:00:25.632395 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:25.632686 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:25.633842 12767 hybrid_clock.cc:648] HybridClock initialized: now 1790492425633818 us; error 43 us; skew 500 ppm
I20260927 07:00:25.635107 12767 webserver.cc:460] Webserver started at http://127.3.57.2:38413/ using document root <none> and password file <none>
I20260927 07:00:25.635401 12767 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:25.635462 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:25.635632 12767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:25.636770 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "445cb746417f46738e057af4e8eab23a"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "9bd075bd3c5ae6ff72e8af414c6fe605"
server_key_iv: "f3c2f2a85c5fc48ce80ba85ffe5af9e3"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.637137 12767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "445cb746417f46738e057af4e8eab23a"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "9bd075bd3c5ae6ff72e8af414c6fe605"
server_key_iv: "f3c2f2a85c5fc48ce80ba85ffe5af9e3"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.638442 12767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:00:25.639178 12781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.639365 12767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:25.639434 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "445cb746417f46738e057af4e8eab23a"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "9bd075bd3c5ae6ff72e8af414c6fe605"
server_key_iv: "f3c2f2a85c5fc48ce80ba85ffe5af9e3"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.639518 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:25.656020 12767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:25.656327 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:25.656471 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:25.656695 12767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:25.657011 12767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:25.657045 12767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.657066 12767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:25.657107 12767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.663065 12767 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:42401
I20260927 07:00:25.663106 12894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:42401 every 8 connection(s)
I20260927 07:00:25.663442 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:25.668316 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.664185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:40167 (local address 127.3.57.62:34909)
0927 07:00:25.664257 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.664259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.664510 (+ 251us) server_negotiation.cc:407] Connection header received
0927 07:00:25.664981 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.664982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.664992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.665053 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.665653 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.666136 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.667048 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.667229 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.667293 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.668083 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:25.668084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:25.668084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:25.668088 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:25.668101 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:25.668105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:25.668147 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:25.668230 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:00:25.668255 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:00:25.668509 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.664164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34909 (local address 127.3.57.2:40167)
0927 07:00:25.664370 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:25.664412 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:25.664899 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:25.665104 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:25.665107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:25.665121 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:25.665582 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.665585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.666215 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.666216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.666964 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.666966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.667315 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.667316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.667417 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.667958 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:25.667958 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:25.667984 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:25.668004 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:25.668158 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:25.668162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:25.668164 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:00:25.668214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:00:25.668289 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260927 07:00:25.669035 12895 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34909
I20260927 07:00:25.669126 12895 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:25.669265 12895 heartbeater.cc:511] Master 127.3.57.62:34909 requested a full tablet report, sending...
I20260927 07:00:25.669575 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12767
I20260927 07:00:25.669651 12576 ts_manager.cc:194] Registered new tserver with Master: 445cb746417f46738e057af4e8eab23a (127.3.57.2:42401)
I20260927 07:00:25.669673 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:00:25.669878 3300 external_mini_cluster.cc:1428] Setting key b1fa5f971670ccd558c2856b6645cc2f
I20260927 07:00:25.670118 12576 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.2:40167
I20260927 07:00:25.670970 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--builtin_ntp_servers=127.3.57.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 07:00:25.753289 12898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:25.753482 12898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:25.753504 12898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:25.754576 12898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:25.754611 12898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:25.754634 12898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:25.755937 12898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:25.756109 12898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:25.756336 12898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:25.760953 12906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.760973 12903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.760953 12904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.761224 12898 server_base.cc:1034] running on GCE node
I20260927 07:00:25.761390 12898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:25.761566 12898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:25.762728 12898 hybrid_clock.cc:648] HybridClock initialized: now 1790492425762722 us; error 28 us; skew 500 ppm
I20260927 07:00:25.763933 12898 webserver.cc:460] Webserver started at http://127.3.57.3:42769/ using document root <none> and password file <none>
I20260927 07:00:25.764155 12898 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:25.764201 12898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:25.764326 12898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:25.765347 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "5fa05db9c47e488ebf9ecc8bb4f23c4c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "a73bfdf36457f52dab0648a4f6e49399"
server_key_iv: "b8e09374800d1e513bb95b826ec19311"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.765695 12898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "5fa05db9c47e488ebf9ecc8bb4f23c4c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "a73bfdf36457f52dab0648a4f6e49399"
server_key_iv: "b8e09374800d1e513bb95b826ec19311"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.766942 12898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:00:25.767735 12912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.767908 12898 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:25.768007 12898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "5fa05db9c47e488ebf9ecc8bb4f23c4c"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "a73bfdf36457f52dab0648a4f6e49399"
server_key_iv: "b8e09374800d1e513bb95b826ec19311"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.768085 12898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:25.798936 12898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:25.799238 12898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:25.799386 12898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:25.799638 12898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:25.799964 12898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:25.800016 12898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.800050 12898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:25.800076 12898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.805382 12898 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:37675
I20260927 07:00:25.805429 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:37675 every 8 connection(s)
I20260927 07:00:25.805728 12898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:00:25.810678 12615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.806405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:43757 (local address 127.3.57.62:34909)
0927 07:00:25.806481 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.806482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.806688 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:00:25.807150 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.807151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.807160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.807213 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.807962 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.808436 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.809407 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.809584 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.809636 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.810388 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:25.810389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:25.810389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:25.810393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:25.810407 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:25.810411 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:25.810456 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:25.810599 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 07:00:25.810619 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 07:00:25.810907 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.806373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34909 (local address 127.3.57.3:43757)
0927 07:00:25.806563 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:25.806602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:25.807071 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:25.807287 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:25.807290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:25.807298 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:25.807887 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.807893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.808515 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.808515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.809332 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:25.809335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:25.809657 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:25.809658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:25.809761 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:25.810243 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:25.810244 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:25.810271 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:25.810302 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:25.810511 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:25.810515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:25.810517 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:00:25.810577 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:00:25.810662 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260927 07:00:25.811436 13026 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34909
I20260927 07:00:25.811532 13026 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:25.811730 13026 heartbeater.cc:511] Master 127.3.57.62:34909 requested a full tablet report, sending...
I20260927 07:00:25.812098 12575 ts_manager.cc:194] Registered new tserver with Master: 5fa05db9c47e488ebf9ecc8bb4f23c4c (127.3.57.3:37675)
I20260927 07:00:25.812472 12575 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.3:43757
I20260927 07:00:25.815694 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 12898
I20260927 07:00:25.815768 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:00:25.815968 3300 external_mini_cluster.cc:1428] Setting key 8d11d7d94e7ddf07812c628edcceb9b3
I20260927 07:00:25.816937 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:00:25.818778 12615 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:25.818243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.3.57.62:34909)
0927 07:00:25.818315 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.818318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.818431 (+ 113us) server_negotiation.cc:407] Connection header received
0927 07:00:25.818519 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.818520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.818531 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.818603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.818725 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52054: BlockingRecv error: recv got EOF from 127.0.0.1:52054 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:00:25.818898 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12636
I20260927 07:00:25.823701 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12767
I20260927 07:00:25.828033 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12898
I20260927 07:00:25.832347 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 12545
2026-09-27T07:00:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (626 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T07:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:25Z Disabled control of system clock
I20260927 07:00:25.858464 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34003
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:00:25.939258 13037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:25.939445 13037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:25.939466 13037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:25.940630 13037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:25.940670 13037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:25.940682 13037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:25.940694 13037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:25.940707 13037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:25.942127 13037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:34003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:25.942353 13037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:25.942637 13037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:25.947250 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.947258 13043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:25.947281 13045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:25.947463 13037 server_base.cc:1034] running on GCE node
I20260927 07:00:25.947695 13037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:25.947966 13037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:25.949141 13037 hybrid_clock.cc:648] HybridClock initialized: now 1790492425949118 us; error 39 us; skew 500 ppm
I20260927 07:00:25.950307 13037 webserver.cc:460] Webserver started at http://127.3.57.62:35003/ using document root <none> and password file <none>
I20260927 07:00:25.950565 13037 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:25.950639 13037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:25.950793 13037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:25.951879 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "049b1acbf00a4a339e79a5340a7023a9"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "7d9022683faabad38ba4b173b70f434e"
server_key_iv: "b1688c54242735342284c5ab3b44bd85"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.952244 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "049b1acbf00a4a339e79a5340a7023a9"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "7d9022683faabad38ba4b173b70f434e"
server_key_iv: "b1688c54242735342284c5ab3b44bd85"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.953501 13037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:25.954200 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:25.954382 13037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:25.954476 13037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "049b1acbf00a4a339e79a5340a7023a9"
format_stamp: "Formatted at 2026-09-27 07:00:25 on dist-test-slave-mk3h"
server_key: "7d9022683faabad38ba4b173b70f434e"
server_key_iv: "b1688c54242735342284c5ab3b44bd85"
server_key_version: "encryptionkey@0"
I20260927 07:00:25.954553 13037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:25.957798 13037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:25.958045 13037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:25.958181 13037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:25.961889 13037 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:34003
I20260927 07:00:25.961920 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:34003 every 8 connection(s)
I20260927 07:00:25.962230 13037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:25.962381 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13037
I20260927 07:00:25.962461 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:00:25.962708 3300 external_mini_cluster.cc:1428] Setting key 57ba0842158090f9a18e9b599d256964
I20260927 07:00:25.963011 13104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:25.965061 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:25.963176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.3.57.62:34003)
0927 07:00:25.963360 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:25.963369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:25.963403 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:00:25.963823 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:25.963829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:25.963865 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:25.963974 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:25.964636 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:25.964639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:25.964640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:25.964643 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:25.964678 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:25.964687 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:25.964741 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:25.964825 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 07:00:25.964891 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260927 07:00:25.966288 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9: Bootstrap starting.
I20260927 07:00:25.966833 13104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:25.967113 13104 log.cc:824] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:25.967797 13104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9: No bootstrap required, opened a new log
I20260927 07:00:25.969067 13104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } }
I20260927 07:00:25.969190 13104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:25.969213 13104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049b1acbf00a4a339e79a5340a7023a9, State: Initialized, Role: FOLLOWER
I20260927 07:00:25.969359 13104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } }
I20260927 07:00:25.969467 13104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:25.969514 13104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:25.969564 13104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:25.970036 13104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } }
I20260927 07:00:25.970116 13104 leader_election.cc:302] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 049b1acbf00a4a339e79a5340a7023a9; no voters:
I20260927 07:00:25.970290 13104 leader_election.cc:288] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:00:25.970537 13104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:25.970570 13109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:25.970700 13109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [term 1 LEADER]: Becoming Leader. State: Replica: 049b1acbf00a4a339e79a5340a7023a9, State: Running, Role: LEADER
I20260927 07:00:25.970880 13109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } }
I20260927 07:00:25.971298 13111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 049b1acbf00a4a339e79a5340a7023a9. Latest consensus state: current_term: 1 leader_uuid: "049b1acbf00a4a339e79a5340a7023a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } } }
I20260927 07:00:25.971398 13111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:25.971890 13110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "049b1acbf00a4a339e79a5340a7023a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049b1acbf00a4a339e79a5340a7023a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 34003 } } }
I20260927 07:00:25.972004 13110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:25.971930 13119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:25.972522 13119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:25.973978 13119 catalog_manager.cc:1347] Generated new cluster ID: 15cf03fd5cf6487ba4e30e7bbbb84e31
I20260927 07:00:25.974035 13119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:00:25.993191 13119 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:25.993422 13119 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:26.004366 13119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 049b1acbf00a4a339e79a5340a7023a9: Generated new TSK 0
I20260927 07:00:26.004637 13119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:00:26.007222 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--builtin_ntp_servers=127.3.57.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:00:26.107463 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:26.107741 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:26.107765 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:26.108918 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:26.108954 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:26.108980 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:26.110244 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:26.110473 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:26.110810 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:26.115693 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.115770 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.115679 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:26.115980 13128 server_base.cc:1034] running on GCE node
I20260927 07:00:26.116184 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:26.116405 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:26.117561 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790492426117549 us; error 29 us; skew 500 ppm
I20260927 07:00:26.118750 13128 webserver.cc:460] Webserver started at http://127.3.57.1:46393/ using document root <none> and password file <none>
I20260927 07:00:26.118988 13128 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:26.119071 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:26.119221 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:26.120287 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "6a9e642a3a65448591242e2596a5feca"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "12d23f838219607872d0ad66d3974535"
server_key_iv: "7d24ec7d53df279c50cbbe365c425c38"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.120630 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "6a9e642a3a65448591242e2596a5feca"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "12d23f838219607872d0ad66d3974535"
server_key_iv: "7d24ec7d53df279c50cbbe365c425c38"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.121914 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:26.122648 13142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.122841 13128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:26.122928 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "6a9e642a3a65448591242e2596a5feca"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "12d23f838219607872d0ad66d3974535"
server_key_iv: "7d24ec7d53df279c50cbbe365c425c38"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.123015 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:26.126407 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:26.126667 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:26.126802 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:26.126991 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:26.127310 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:26.127363 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.127401 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:26.127429 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.132789 13128 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44775
I20260927 07:00:26.132848 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44775 every 8 connection(s)
I20260927 07:00:26.133163 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:00:26.133764 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13128
I20260927 07:00:26.133876 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:00:26.134081 3300 external_mini_cluster.cc:1428] Setting key 38f815a9a8334a5258fa874cf9bd6f1f
I20260927 07:00:26.134913 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--builtin_ntp_servers=127.3.57.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 07:00:26.136394 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.133901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:45029 (local address 127.3.57.62:34003)
0927 07:00:26.133956 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:26.133958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:26.134224 (+ 266us) server_negotiation.cc:407] Connection header received
0927 07:00:26.134790 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:26.134791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:26.134798 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:26.134842 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:26.136146 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:26.136147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:26.136147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:26.136151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:26.136168 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:26.136173 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:26.136234 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:26.136330 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:00:26.136343 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:00:26.136618 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.133890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34003 (local address 127.3.57.1:45029)
0927 07:00:26.134096 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:26.134133 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:26.134700 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:26.134903 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:26.134907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:26.134915 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:26.135959 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:26.135960 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:26.135999 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:26.136035 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:26.136249 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:26.136255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:26.136258 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:26.136335 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:00:26.136398 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260927 07:00:26.137104 13256 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34003
I20260927 07:00:26.137198 13256 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:26.137336 13256 heartbeater.cc:511] Master 127.3.57.62:34003 requested a full tablet report, sending...
I20260927 07:00:26.137816 13068 ts_manager.cc:194] Registered new tserver with Master: 6a9e642a3a65448591242e2596a5feca (127.3.57.1:44775)
W20260927 07:00:26.214274 13259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:26.214462 13259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:26.214484 13259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:26.215582 13259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:26.215616 13259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:26.215646 13259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:26.216862 13259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:26.217037 13259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:26.217259 13259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:00:26.222136 13259 server_base.cc:1034] running on GCE node
W20260927 07:00:26.222183 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.222103 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.222095 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:26.222492 13259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:26.222720 13259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:26.223852 13259 hybrid_clock.cc:648] HybridClock initialized: now 1790492426223840 us; error 28 us; skew 500 ppm
I20260927 07:00:26.224980 13259 webserver.cc:460] Webserver started at http://127.3.57.2:39955/ using document root <none> and password file <none>
I20260927 07:00:26.225214 13259 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:26.225284 13259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:26.225463 13259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:26.226568 13259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "293b2695bbec4530bcc1511bff9346e1"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "7f430c561da9623e12095b6c81c2dbf0"
server_key_iv: "ef772e6af9b585c6f754510232592ef5"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.226958 13259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "293b2695bbec4530bcc1511bff9346e1"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "7f430c561da9623e12095b6c81c2dbf0"
server_key_iv: "ef772e6af9b585c6f754510232592ef5"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.228319 13259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:00:26.229009 13273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.229171 13259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:26.229266 13259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "293b2695bbec4530bcc1511bff9346e1"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "7f430c561da9623e12095b6c81c2dbf0"
server_key_iv: "ef772e6af9b585c6f754510232592ef5"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.229351 13259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:26.232659 13259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:26.232913 13259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:26.233057 13259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:26.233268 13259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:26.233551 13259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:26.233605 13259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.233637 13259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:26.233662 13259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.239205 13259 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:45731
I20260927 07:00:26.239240 13386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:45731 every 8 connection(s)
I20260927 07:00:26.239542 13259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:26.239846 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13259
I20260927 07:00:26.239956 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:00:26.240182 3300 external_mini_cluster.cc:1428] Setting key 5569267c3783481438237146abe8f1da
I20260927 07:00:26.241155 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--builtin_ntp_servers=127.3.57.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 07:00:26.242950 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.240502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44907 (local address 127.3.57.62:34003)
0927 07:00:26.240552 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:26.240554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:26.240699 (+ 145us) server_negotiation.cc:407] Connection header received
0927 07:00:26.241215 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:26.241216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:26.241223 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:26.241298 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:26.242702 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:26.242703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:26.242704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:26.242709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:26.242727 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:26.242731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:26.242794 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:26.242849 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 07:00:26.242864 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:00:26.243144 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.240383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34003 (local address 127.3.57.2:44907)
0927 07:00:26.240570 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:26.240607 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:26.241104 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:26.241428 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:26.241432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:26.241441 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:26.242515 (+ 1074us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:26.242516 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:26.242561 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:26.242599 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:26.242795 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:26.242797 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:26.242800 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:26.242853 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:00:26.242899 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260927 07:00:26.243624 13387 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34003
I20260927 07:00:26.243709 13387 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:26.243873 13387 heartbeater.cc:511] Master 127.3.57.62:34003 requested a full tablet report, sending...
I20260927 07:00:26.244227 13068 ts_manager.cc:194] Registered new tserver with Master: 293b2695bbec4530bcc1511bff9346e1 (127.3.57.2:45731)
W20260927 07:00:26.320016 13390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:26.320194 13390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:26.320214 13390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:26.321276 13390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:26.321309 13390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:26.321333 13390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:26.322559 13390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:26.322738 13390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:26.322964 13390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:26.327880 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.327857 13396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:26.327909 13395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:26.327955 13390 server_base.cc:1034] running on GCE node
I20260927 07:00:26.328300 13390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:26.328536 13390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:26.329697 13390 hybrid_clock.cc:648] HybridClock initialized: now 1790492426329685 us; error 30 us; skew 500 ppm
I20260927 07:00:26.330999 13390 webserver.cc:460] Webserver started at http://127.3.57.3:40939/ using document root <none> and password file <none>
I20260927 07:00:26.331259 13390 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:26.331329 13390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:26.331481 13390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:26.332511 13390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "72ec74c0a0244ea6bfb31d1037e2fb4f"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "4cbcbb2d8a2d955e5204186ef2fa0166"
server_key_iv: "2c0b256e2cfd235e7d22dd803d85f705"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.332862 13390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "72ec74c0a0244ea6bfb31d1037e2fb4f"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "4cbcbb2d8a2d955e5204186ef2fa0166"
server_key_iv: "2c0b256e2cfd235e7d22dd803d85f705"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.334103 13390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:00:26.334802 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.334968 13390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:26.335062 13390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "72ec74c0a0244ea6bfb31d1037e2fb4f"
format_stamp: "Formatted at 2026-09-27 07:00:26 on dist-test-slave-mk3h"
server_key: "4cbcbb2d8a2d955e5204186ef2fa0166"
server_key_iv: "2c0b256e2cfd235e7d22dd803d85f705"
server_key_version: "encryptionkey@0"
I20260927 07:00:26.335143 13390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:26.338292 13390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:26.338527 13390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:26.338656 13390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:26.338851 13390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:26.339129 13390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:26.339198 13390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.339236 13390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:26.339262 13390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:26.344888 13390 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:38355
I20260927 07:00:26.344923 13517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:38355 every 8 connection(s)
I20260927 07:00:26.345221 13390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:00:26.346408 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13390
I20260927 07:00:26.346484 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:00:26.346685 3300 external_mini_cluster.cc:1428] Setting key 66969107a007bf74782e3244d8d02b4c
I20260927 07:00:26.348251 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.345879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:35613 (local address 127.3.57.62:34003)
0927 07:00:26.345923 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:26.345925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:26.346165 (+ 240us) server_negotiation.cc:407] Connection header received
0927 07:00:26.346739 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:26.346740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:26.346747 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:26.346797 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:26.347990 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:26.347991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:26.347991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:00:26.347996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:26.348011 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:00:26.348015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:26.348090 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:26.348168 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 07:00:26.348182 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:00:26.348428 13520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:26.345876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:34003 (local address 127.3.57.3:35613)
0927 07:00:26.346051 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:26.346082 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:26.346633 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:26.346860 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:26.346863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:26.346872 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:26.347847 (+ 975us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:00:26.347847 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:26.347872 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:00:26.347898 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:26.348092 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:26.348097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:26.348100 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:26.348167 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 07:00:26.348224 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260927 07:00:26.348939 13518 heartbeater.cc:347] Connected to a master server at 127.3.57.62:34003
I20260927 07:00:26.349030 13518 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:26.349128 13518 heartbeater.cc:511] Master 127.3.57.62:34003 requested a full tablet report, sending...
I20260927 07:00:26.349429 13068 ts_manager.cc:194] Registered new tserver with Master: 72ec74c0a0244ea6bfb31d1037e2fb4f (127.3.57.3:38355)
I20260927 07:00:26.358596 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:00:26.360351 13107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:00:26.359891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.3.57.62:34003)
0927 07:00:26.359953 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:26.359955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:26.360076 (+ 121us) server_negotiation.cc:407] Connection header received
0927 07:00:26.360103 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:26.360104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:26.360109 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:26.360158 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:26.360289 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56028: BlockingRecv error: recv got EOF from 127.0.0.1:56028 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:00:26.360455 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13128
I20260927 07:00:26.365408 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13259
I20260927 07:00:26.369638 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13390
I20260927 07:00:26.373885 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13037
2026-09-27T07:00:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (541 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:00:26 dist-test-slave-mk3h krb5kdc[13528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:00:26 dist-test-slave-mk3h krb5kdc[13528](info): set up 2 sockets
Sep 27 07:00:26 dist-test-slave-mk3h krb5kdc[13528](info): commencing operation
krb5kdc: starting...
W20260927 07:00:27.407892 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:00:27.522745 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:33713
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:33713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:27.600019 13541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:27.600195 13541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:27.600216 13541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:27.601354 13541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:00:27.601395 13541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:27.601410 13541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:27.601447 13541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:00:27.601476 13541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:00:27.602890 13541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:33713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:27.603125 13541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:27.603353 13541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:27.608008 13547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:27.608052 13541 server_base.cc:1034] running on GCE node
W20260927 07:00:27.608018 13549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:27.608024 13546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:27.608405 13541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:27.608700 13541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:27.609861 13541 hybrid_clock.cc:648] HybridClock initialized: now 1790492427609853 us; error 42 us; skew 500 ppm
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:27.612196 13541 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:00:27.612636 13541 webserver.cc:460] Webserver started at http://127.3.57.62:36435/ using document root <none> and password file <none>
I20260927 07:00:27.612875 13541 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:27.612941 13541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:27.613089 13541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:27.614053 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "ae1bb2c9119d400bbf53454b9fe23599"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "df4396213ce3859199f02e0d84c83e9b"
server_key_iv: "2665b1c1e63979312645af4845b3f64e"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.614396 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "ae1bb2c9119d400bbf53454b9fe23599"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "df4396213ce3859199f02e0d84c83e9b"
server_key_iv: "2665b1c1e63979312645af4845b3f64e"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.615733 13541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:00:27.616426 13556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.616629 13541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:27.616711 13541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "ae1bb2c9119d400bbf53454b9fe23599"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "df4396213ce3859199f02e0d84c83e9b"
server_key_iv: "2665b1c1e63979312645af4845b3f64e"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.616770 13541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:27.650431 13541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:27.652290 13541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:27.652396 13541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:27.656278 13541 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:33713
I20260927 07:00:27.656351 13608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:33713 every 8 connection(s)
I20260927 07:00:27.656667 13541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:00:27.656939 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13541
I20260927 07:00:27.657040 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:00:27.657239 3300 external_mini_cluster.cc:1428] Setting key f569bc0b16c9afbbb3da0427aee214b1
I20260927 07:00:27.657892 13609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:27.660873 13609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: Bootstrap starting.
I20260927 07:00:27.661553 13609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:27.661845 13609 log.cc:824] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:27.662735 13609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: No bootstrap required, opened a new log
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:27.664929 13609 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } }
I20260927 07:00:27.665124 13609 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:27.665174 13609 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1bb2c9119d400bbf53454b9fe23599, State: Initialized, Role: FOLLOWER
I20260927 07:00:27.665313 13609 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } }
I20260927 07:00:27.665441 13609 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:00:27.665481 13609 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:00:27.665553 13609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:27.666268 13609 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } }
I20260927 07:00:27.666386 13609 leader_election.cc:302] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae1bb2c9119d400bbf53454b9fe23599; no voters:
I20260927 07:00:27.666591 13609 leader_election.cc:288] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:00:27.666656 13614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:27.666823 13614 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [term 1 LEADER]: Becoming Leader. State: Replica: ae1bb2c9119d400bbf53454b9fe23599, State: Running, Role: LEADER
I20260927 07:00:27.666859 13609 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:00:27.667001 13614 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } }
I20260927 07:00:27.667457 13614 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae1bb2c9119d400bbf53454b9fe23599. Latest consensus state: current_term: 1 leader_uuid: "ae1bb2c9119d400bbf53454b9fe23599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } } }
I20260927 07:00:27.667721 13614 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:27.667637 13615 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae1bb2c9119d400bbf53454b9fe23599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae1bb2c9119d400bbf53454b9fe23599" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33713 } } }
I20260927 07:00:27.668089 13615 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599 [sys.catalog]: This master's current role is: LEADER
I20260927 07:00:27.668203 13623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:00:27.668726 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:27.657750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.3.57.62:33713)
0927 07:00:27.657938 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:27.657948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:27.657979 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:00:27.658547 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:27.658556 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:27.658614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:27.658766 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:27.659296 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.660607 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.661357 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.661974 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.662087 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:27.664248 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:27.664252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:27.664253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:27.664265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:27.666482 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.666899 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.666902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.666902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.666996 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.667255 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.667256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.667256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.667610 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:27.667784 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:27.668286 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 07:00:27.668465 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260927 07:00:27.668768 13623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:00:27.670224 13623 catalog_manager.cc:1347] Generated new cluster ID: 5b229e485bc04d17b0dffa8d72f69d89
I20260927 07:00:27.670277 13623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:00:27.682287 13623 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:00:27.682946 13623 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:00:27.687665 13623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: Generated new TSK 0
I20260927 07:00:27.687819 13623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:00:27.714144 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:27.796684 13635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:27.796900 13635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:27.796936 13635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:27.798161 13635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:27.798216 13635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:27.798264 13635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:00:27.799930 13635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:27.800206 13635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:27.800453 13635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:27.805222 13640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:27.805229 13641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:27.805229 13643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:27.805310 13635 server_base.cc:1034] running on GCE node
I20260927 07:00:27.805612 13635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:27.805884 13635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:27.807044 13635 hybrid_clock.cc:648] HybridClock initialized: now 1790492427807031 us; error 41 us; skew 500 ppm
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:27.809541 13635 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:00:27.809971 13635 webserver.cc:460] Webserver started at http://127.3.57.1:33629/ using document root <none> and password file <none>
I20260927 07:00:27.810197 13635 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:27.810264 13635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:27.810408 13635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:27.811421 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "08eb48c6f28141c8b189b4e8e00f7ddc"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "6ff5f7c74b4670cdca2bc65ce0b9fb40"
server_key_iv: "686ded0dbcfd9386e14d3a2df17620b8"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.811802 13635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "08eb48c6f28141c8b189b4e8e00f7ddc"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "6ff5f7c74b4670cdca2bc65ce0b9fb40"
server_key_iv: "686ded0dbcfd9386e14d3a2df17620b8"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.813048 13635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:00:27.813717 13650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.813918 13635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:00:27.814015 13635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "08eb48c6f28141c8b189b4e8e00f7ddc"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "6ff5f7c74b4670cdca2bc65ce0b9fb40"
server_key_iv: "686ded0dbcfd9386e14d3a2df17620b8"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.814095 13635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:27.821830 13635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:27.823964 13635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:27.824118 13635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:27.824335 13635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:27.824656 13635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:27.824714 13635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.824748 13635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:27.824775 13635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.830477 13635 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:40223
I20260927 07:00:27.830554 13763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:40223 every 8 connection(s)
I20260927 07:00:27.830860 13635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:27.838635 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13635
I20260927 07:00:27.838757 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:00:27.838966 3300 external_mini_cluster.cc:1428] Setting key 45dfdded616c5ae7e001ec76ca93d16a
I20260927 07:00:27.839324 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:27.831540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:36103 (local address 127.3.57.62:33713)
0927 07:00:27.831639 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:27.831641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:27.831799 (+ 158us) server_negotiation.cc:407] Connection header received
0927 07:00:27.832232 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:27.832233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:27.832242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:27.832293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:27.833134 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.833618 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.834533 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.834728 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.834778 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:27.836734 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:27.836735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:27.836736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:27.836742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:27.838072 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.838395 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.838396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.838396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.838443 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.838679 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.838680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.838680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.838820 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:27.838901 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:27.839145 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 07:00:27.839221 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:00:27.839493 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:27.831523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:33713 (local address 127.3.57.1:36103)
0927 07:00:27.831679 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:27.831712 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:27.832157 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:27.832342 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:27.832344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:27.832884 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:27.833058 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:27.833062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.833700 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.833700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:27.834461 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:27.834463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.834804 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.834804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:27.834897 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:27.835305 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:27.835312 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:27.836655 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:27.838138 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:27.838143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:27.838144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:27.838334 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:27.838528 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:27.838529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:27.838529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:27.838615 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:27.838890 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:27.838895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:27.839000 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 07:00:27.839111 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:00:27.839278 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260927 07:00:27.839937 13764 heartbeater.cc:347] Connected to a master server at 127.3.57.62:33713
I20260927 07:00:27.840039 13764 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:27.840198 13764 heartbeater.cc:511] Master 127.3.57.62:33713 requested a full tablet report, sending...
I20260927 07:00:27.840691 13573 ts_manager.cc:194] Registered new tserver with Master: 08eb48c6f28141c8b189b4e8e00f7ddc (127.3.57.1:40223)
I20260927 07:00:27.841266 13573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:36103
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:00:27.862596 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:27.942927 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:27.943107 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:27.943128 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:27.944268 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:27.944308 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:27.944334 13769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:00:27.945721 13769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:27.945914 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:27.946130 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:27.950834 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:27.950937 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:27.950986 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:27.951179 13769 server_base.cc:1034] running on GCE node
I20260927 07:00:27.951472 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:27.951747 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:27.952917 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790492427952905 us; error 24 us; skew 500 ppm
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:27.955199 13769 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:00:27.955708 13769 webserver.cc:460] Webserver started at http://127.3.57.2:33605/ using document root <none> and password file <none>
I20260927 07:00:27.955984 13769 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:27.956064 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:27.956189 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:27.957198 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "bd104521d3b64acf848c7bad7918e701"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "3ceea601cb27be86aa59367112491af9"
server_key_iv: "a4b855e34b1afc94e291ddfc76a71125"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.957572 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "bd104521d3b64acf848c7bad7918e701"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "3ceea601cb27be86aa59367112491af9"
server_key_iv: "a4b855e34b1afc94e291ddfc76a71125"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.958802 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:00:27.959476 13784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.959635 13769 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.959722 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "bd104521d3b64acf848c7bad7918e701"
format_stamp: "Formatted at 2026-09-27 07:00:27 on dist-test-slave-mk3h"
server_key: "3ceea601cb27be86aa59367112491af9"
server_key_iv: "a4b855e34b1afc94e291ddfc76a71125"
server_key_version: "encryptionkey@0"
I20260927 07:00:27.959774 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:27.976750 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:27.978391 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:27.978494 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:27.978657 13769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:27.978883 13769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:27.978924 13769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.978945 13769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:27.978956 13769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:27.984441 13769 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:39207
I20260927 07:00:27.984513 13897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:39207 every 8 connection(s)
I20260927 07:00:27.984817 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:00:27.986753 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13769
I20260927 07:00:27.986831 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:00:27.987031 3300 external_mini_cluster.cc:1428] Setting key 16c48c2be10d94ac80731c5b386330d3
Sep 27 07:00:27 dist-test-slave-mk3h krb5kdc[13528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:27.994558 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:27.985535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44779 (local address 127.3.57.62:33713)
0927 07:00:27.985579 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:27.985581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:27.985771 (+ 190us) server_negotiation.cc:407] Connection header received
0927 07:00:27.986218 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:27.986219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:27.986228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:27.986281 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:27.987351 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.987921 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.988971 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.989197 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.989255 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:27.991789 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:27.991790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:27.991791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:27.991797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:27.993211 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.993658 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.993659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.993659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.993706 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:27.993984 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:27.993984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:27.993984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:27.994070 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:27.994115 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:27.994441 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:00:27.994497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:27.994799 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:27.985484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:33713 (local address 127.3.57.2:44779)
0927 07:00:27.985642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:27.985680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:27.986138 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:27.986330 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:27.986333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:27.986976 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:27.987229 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:27.987235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.988010 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.988011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:27.988862 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:27.988865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:27.989283 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:27.989284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:27.989427 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:27.989913 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:27.989924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:27.991699 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:27.993291 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:27.993298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:27.993300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:27.993586 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:27.993796 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:27.993797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:27.993797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:27.993924 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:27.994138 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:27.994143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:27.994288 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 07:00:27.994423 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 07:00:27.994572 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260927 07:00:27.995309 13898 heartbeater.cc:347] Connected to a master server at 127.3.57.62:33713
I20260927 07:00:27.995397 13898 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:27.995615 13898 heartbeater.cc:511] Master 127.3.57.62:33713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:00:27.996049 13573 ts_manager.cc:194] Registered new tserver with Master: bd104521d3b64acf848c7bad7918e701 (127.3.57.2:39207)
I20260927 07:00:27.996405 13573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:44779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:00:28.012378 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:00:28.090785 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:00:28.091001 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:00:28.091035 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:00:28.092228 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:00:28.092291 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:00:28.092338 13903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:00:28.093739 13903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:33713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:00:28.093982 13903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:00:28.094197 13903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:00:28.098933 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:00:28.098953 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:28.099017 13903 server_base.cc:1034] running on GCE node
W20260927 07:00:28.098976 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:00:28.099303 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:00:28.099628 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:00:28.100781 13903 hybrid_clock.cc:648] HybridClock initialized: now 1790492428100763 us; error 29 us; skew 500 ppm
Sep 27 07:00:28 dist-test-slave-mk3h krb5kdc[13528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:00:28.103091 13903 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:00:28.103477 13903 webserver.cc:460] Webserver started at http://127.3.57.3:36857/ using document root <none> and password file <none>
I20260927 07:00:28.103749 13903 fs_manager.cc:359] Metadata directory not provided
I20260927 07:00:28.103816 13903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:00:28.103961 13903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:00:28.104915 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "3aa727f601494e168d692bd3c4859262"
format_stamp: "Formatted at 2026-09-27 07:00:28 on dist-test-slave-mk3h"
server_key: "ad5745b07c257150295d4774d9080f8f"
server_key_iv: "e73139891fb7b0ebe4656ce4f95cf52d"
server_key_version: "encryptionkey@0"
I20260927 07:00:28.105269 13903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "3aa727f601494e168d692bd3c4859262"
format_stamp: "Formatted at 2026-09-27 07:00:28 on dist-test-slave-mk3h"
server_key: "ad5745b07c257150295d4774d9080f8f"
server_key_iv: "e73139891fb7b0ebe4656ce4f95cf52d"
server_key_version: "encryptionkey@0"
I20260927 07:00:28.106592 13903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:00:28.107335 13918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:28.107506 13903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:00:28.107620 13903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "3aa727f601494e168d692bd3c4859262"
format_stamp: "Formatted at 2026-09-27 07:00:28 on dist-test-slave-mk3h"
server_key: "ad5745b07c257150295d4774d9080f8f"
server_key_iv: "e73139891fb7b0ebe4656ce4f95cf52d"
server_key_version: "encryptionkey@0"
I20260927 07:00:28.107712 13903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:00:28.137660 13903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:00:28.139514 13903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:00:28.139700 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:00:28.139930 13903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:00:28.140255 13903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:00:28.140309 13903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:28.140342 13903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:00:28.140369 13903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:00:28.145866 13903 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:45559
I20260927 07:00:28.145924 14031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:45559 every 8 connection(s)
I20260927 07:00:28.146214 13903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:00:28.146844 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 13903
I20260927 07:00:28.146941 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:00:28.147179 3300 external_mini_cluster.cc:1428] Setting key 877d6f9a560f5b7a03776d5ef32225a5
Sep 27 07:00:28 dist-test-slave-mk3h krb5kdc[13528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:00:28.155153 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.146920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:49273 (local address 127.3.57.62:33713)
0927 07:00:28.146976 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.146979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.147206 (+ 227us) server_negotiation.cc:407] Connection header received
0927 07:00:28.147943 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.147944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.147956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.148015 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:28.148971 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.149460 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.150393 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.150583 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.150634 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.152689 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:28.152690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:28.152690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:28.152696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:28.154068 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:28.154414 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:28.154415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:28.154415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:28.154469 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:28.154678 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:28.154678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:28.154679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:28.154772 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:28.154823 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.155033 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.155090 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:00:28.155301 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.146874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:33713 (local address 127.3.57.3:49273)
0927 07:00:28.147064 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.147113 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.147814 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.148097 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.148101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.148698 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:00:28.148874 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.148879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.149535 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.149536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.150323 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.150325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.150663 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.150663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.150757 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.151241 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:00:28.151257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:00:28.152606 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:00:28.154139 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:28.154144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:28.154145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:28.154345 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:28.154530 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:00:28.154530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:00:28.154530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:00:28.154619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:00:28.154825 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:00:28.154829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:00:28.154932 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.155007 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.155115 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260927 07:00:28.155805 14032 heartbeater.cc:347] Connected to a master server at 127.3.57.62:33713
I20260927 07:00:28.155906 14032 heartbeater.cc:464] Registering TS with master...
I20260927 07:00:28.156085 14032 heartbeater.cc:511] Master 127.3.57.62:33713 requested a full tablet report, sending...
I20260927 07:00:28.156479 13573 ts_manager.cc:194] Registered new tserver with Master: 3aa727f601494e168d692bd3c4859262 (127.3.57.3:45559)
I20260927 07:00:28.156845 13573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:49273
I20260927 07:00:28.158944 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:00:28.165814 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.160237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.3.57.62:33713)
0927 07:00:28.160281 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.160282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.160377 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:00:28.160439 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.160440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.160447 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.160487 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:00:28.160996 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.161445 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.162234 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.162414 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.162464 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.163834 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:00:28.163835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:00:28.163835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:00:28.163841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:00:28.164928 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:28.165238 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:28.165239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:28.165239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:28.165287 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:00:28.165414 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:00:28.165415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:00:28.165415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:00:28.165535 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:00:28.165609 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.165696 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.165754 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:00:28.167383 13573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:00:28.169229 13573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:00:28.175485 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:45559 (local address 127.0.0.1:59964)
0927 07:00:28.171471 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.171508 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.171591 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.172046 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.172048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.172050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:28.172105 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.172106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173439 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.173440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.174569 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.174571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175191 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.175272 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175286 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.175351 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.175399 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":29,"thread_start_us":51,"threads_started":1}
I20260927 07:00:28.175586 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.3.57.1:40223)
0927 07:00:28.171788 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.171793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.171805 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:00:28.171844 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.171846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.171862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.171941 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:28.172137 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173293 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.174391 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175104 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175164 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.175448 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.175486 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260927 07:00:28.175810 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.3.57.2:39207)
0927 07:00:28.172074 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.172080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.172090 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:28.172126 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.172128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.172143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.172224 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:28.172380 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173126 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.174560 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175236 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175331 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175384 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.175684 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.175722 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":85,"threads_started":1}
I20260927 07:00:28.175822 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.3.57.3:45559)
0927 07:00:28.171872 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.171875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.171883 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:28.171929 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.171931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.171947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.172029 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:28.172169 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173342 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.174634 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175101 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175190 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175251 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.175715 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.175767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":326,"threads_started":1}
I20260927 07:00:28.175638 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:39207 (local address 127.0.0.1:45588)
0927 07:00:28.171580 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.171591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.171644 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.172221 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.172221 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.172223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:28.172266 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.172267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173227 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.173228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.174471 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.174474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175336 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.175392 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175395 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.175448 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.175484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":56,"threads_started":1}
I20260927 07:00:28.175822 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.171331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40223 (local address 127.0.0.1:45934)
0927 07:00:28.171704 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.171714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.171750 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.171962 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.171964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.171967 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:28.172030 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.172033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.173412 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.173413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.174298 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.174301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.175283 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.175284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.175345 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.175350 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.175426 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.175468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":29,"thread_start_us":44,"threads_started":1}
I20260927 07:00:28.176653 13966 tablet_service.cc:1467] Processing CreateTablet for tablet d570753df2a44512a0f3d6391804e3de (DEFAULT_TABLE table=test-table [id=c7e155ac7ad3496ba2929c3dcf7c456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:28.176724 13832 tablet_service.cc:1467] Processing CreateTablet for tablet d570753df2a44512a0f3d6391804e3de (DEFAULT_TABLE table=test-table [id=c7e155ac7ad3496ba2929c3dcf7c456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:28.176852 13698 tablet_service.cc:1467] Processing CreateTablet for tablet d570753df2a44512a0f3d6391804e3de (DEFAULT_TABLE table=test-table [id=c7e155ac7ad3496ba2929c3dcf7c456c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:00:28.177009 13966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d570753df2a44512a0f3d6391804e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:28.177057 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d570753df2a44512a0f3d6391804e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:28.177143 13698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d570753df2a44512a0f3d6391804e3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:00:28.179430 14046 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Bootstrap starting.
I20260927 07:00:28.180088 14046 tablet_bootstrap.cc:654] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:28.180364 14047 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Bootstrap starting.
I20260927 07:00:28.180416 14048 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Bootstrap starting.
I20260927 07:00:28.180819 14046 log.cc:824] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:28.180998 14048 tablet_bootstrap.cc:654] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:28.181046 14047 tablet_bootstrap.cc:654] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Neither blocks nor log segments found. Creating new log.
I20260927 07:00:28.181556 14048 log.cc:824] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:28.181581 14047 log.cc:824] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Log is configured to *not* fsync() on all Append() calls
I20260927 07:00:28.181689 14046 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: No bootstrap required, opened a new log
I20260927 07:00:28.181772 14046 ts_tablet_manager.cc:1397] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:00:28.182227 14047 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: No bootstrap required, opened a new log
I20260927 07:00:28.182313 14047 ts_tablet_manager.cc:1397] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:00:28.183372 14046 raft_consensus.cc:348] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.183508 14046 raft_consensus.cc:374] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:28.183530 14046 raft_consensus.cc:729] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd104521d3b64acf848c7bad7918e701, State: Initialized, Role: FOLLOWER
I20260927 07:00:28.183686 14046 consensus_queue.cc:260] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.183977 14046 ts_tablet_manager.cc:1428] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:00:28.183998 13898 heartbeater.cc:503] Master 127.3.57.62:33713 was elected leader, sending a full tablet report...
I20260927 07:00:28.184087 14047 raft_consensus.cc:348] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.184238 14047 raft_consensus.cc:374] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:28.184280 14047 raft_consensus.cc:729] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa727f601494e168d692bd3c4859262, State: Initialized, Role: FOLLOWER
I20260927 07:00:28.184394 14047 consensus_queue.cc:260] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.184546 14048 tablet_bootstrap.cc:492] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: No bootstrap required, opened a new log
I20260927 07:00:28.184621 14048 ts_tablet_manager.cc:1397] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 07:00:28.184685 14047 ts_tablet_manager.cc:1428] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:00:28.184741 14032 heartbeater.cc:503] Master 127.3.57.62:33713 was elected leader, sending a full tablet report...
I20260927 07:00:28.186230 14048 raft_consensus.cc:348] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.186347 14048 raft_consensus.cc:374] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:00:28.186368 14048 raft_consensus.cc:729] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08eb48c6f28141c8b189b4e8e00f7ddc, State: Initialized, Role: FOLLOWER
I20260927 07:00:28.186439 14048 consensus_queue.cc:260] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.186707 14048 ts_tablet_manager.cc:1428] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:00:28.186724 13764 heartbeater.cc:503] Master 127.3.57.62:33713 was elected leader, sending a full tablet report...
W20260927 07:00:28.235391 13899 tablet.cc:2369] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:00:28.331605 13765 tablet.cc:2369] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:00:28.396766 14033 tablet.cc:2369] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:00:28.401507 14052 raft_consensus.cc:482] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:00:28.401605 14052 raft_consensus.cc:504] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.401959 14052 leader_election.cc:288] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aa727f601494e168d692bd3c4859262 (127.3.57.3:45559), 08eb48c6f28141c8b189b4e8e00f7ddc (127.3.57.1:40223)
I20260927 07:00:28.404814 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.402096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:45559 (local address 127.3.57.2:46563)
0927 07:00:28.402177 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.402190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.402237 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.402427 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.402428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.402429 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:28.402505 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.402507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.403143 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.403144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.403933 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.403935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.404506 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404527 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.404606 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.404609 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.404686 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.404743 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 07:00:28.404841 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.402257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:46563 (local address 127.3.57.3:45559)
0927 07:00:28.402301 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.402303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.402311 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:00:28.402335 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.402336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.402354 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.402422 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:28.402580 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.403058 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404030 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.404411 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404528 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.404539 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.404722 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.404781 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:00:28.405076 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.402211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:40815 (local address 127.3.57.1:40223)
0927 07:00:28.402252 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.402255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.402410 (+ 155us) server_negotiation.cc:407] Connection header received
0927 07:00:28.402482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.402483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.402501 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.402569 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:00:28.402721 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.403173 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404256 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.404648 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.404766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.404943 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.405001 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:00:28.405180 13986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570753df2a44512a0f3d6391804e3de" candidate_uuid: "bd104521d3b64acf848c7bad7918e701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa727f601494e168d692bd3c4859262" is_pre_election: true
I20260927 07:00:28.405268 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.402157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40223 (local address 127.3.57.2:40815)
0927 07:00:28.402329 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.402338 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:00:28.402418 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:00:28.402574 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:00:28.402575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:00:28.402576 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:00:28.402649 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.402651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.403268 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.403269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.404163 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:00:28.404166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.404760 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.404761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:00:28.404827 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.404831 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:00:28.405155 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 07:00:28.405196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260927 07:00:28.405370 13986 raft_consensus.cc:2454] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd104521d3b64acf848c7bad7918e701 in term 0.
I20260927 07:00:28.405580 13787 leader_election.cc:302] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa727f601494e168d692bd3c4859262, bd104521d3b64acf848c7bad7918e701; no voters:
I20260927 07:00:28.405586 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570753df2a44512a0f3d6391804e3de" candidate_uuid: "bd104521d3b64acf848c7bad7918e701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" is_pre_election: true
I20260927 07:00:28.405721 13718 raft_consensus.cc:2454] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd104521d3b64acf848c7bad7918e701 in term 0.
I20260927 07:00:28.405714 14052 raft_consensus.cc:2781] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:00:28.405781 14052 raft_consensus.cc:482] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:00:28.405810 14052 raft_consensus.cc:3037] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:28.406534 14052 raft_consensus.cc:504] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.406687 14052 leader_election.cc:288] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [CANDIDATE]: Term 1 election: Requested vote from peers 3aa727f601494e168d692bd3c4859262 (127.3.57.3:45559), 08eb48c6f28141c8b189b4e8e00f7ddc (127.3.57.1:40223)
I20260927 07:00:28.406852 13718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570753df2a44512a0f3d6391804e3de" candidate_uuid: "bd104521d3b64acf848c7bad7918e701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc"
I20260927 07:00:28.406852 13986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d570753df2a44512a0f3d6391804e3de" candidate_uuid: "bd104521d3b64acf848c7bad7918e701" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa727f601494e168d692bd3c4859262"
I20260927 07:00:28.406929 13986 raft_consensus.cc:3037] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:28.406929 13718 raft_consensus.cc:3037] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:00:28.407824 13986 raft_consensus.cc:2454] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd104521d3b64acf848c7bad7918e701 in term 1.
I20260927 07:00:28.407824 13718 raft_consensus.cc:2454] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd104521d3b64acf848c7bad7918e701 in term 1.
I20260927 07:00:28.408001 13787 leader_election.cc:302] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa727f601494e168d692bd3c4859262, bd104521d3b64acf848c7bad7918e701; no voters:
I20260927 07:00:28.408118 14052 raft_consensus.cc:2781] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:00:28.408263 14052 raft_consensus.cc:686] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 1 LEADER]: Becoming Leader. State: Replica: bd104521d3b64acf848c7bad7918e701, State: Running, Role: LEADER
I20260927 07:00:28.408360 14052 consensus_queue.cc:237] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } }
I20260927 07:00:28.409251 13573 catalog_manager.cc:5495] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd104521d3b64acf848c7bad7918e701 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "bd104521d3b64acf848c7bad7918e701" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd104521d3b64acf848c7bad7918e701" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 } health_report { overall_health: UNKNOWN } } }
I20260927 07:00:28.460927 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.458275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.3.57.2:39207)
0927 07:00:28.458341 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:00:28.458343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:00:28.458353 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:00:28.458429 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:00:28.458434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:00:28.458454 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:00:28.458529 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:00:28.458679 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.459250 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.460106 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:00:28.460324 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:00:28.460406 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:00:28.460560 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:00:28.460668 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:00:28.460764 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:00:28.460789 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:00:28.460809 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:00:28.462953 13718 raft_consensus.cc:1261] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 1 FOLLOWER]: Refusing update from remote peer bd104521d3b64acf848c7bad7918e701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:28.462953 13986 raft_consensus.cc:1261] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 1 FOLLOWER]: Refusing update from remote peer bd104521d3b64acf848c7bad7918e701: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:00:28.463367 14056 consensus_queue.cc:1035] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa727f601494e168d692bd3c4859262" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:28.463512 14052 consensus_queue.cc:1035] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08eb48c6f28141c8b189b4e8e00f7ddc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:00:28.466207 14061 mvcc.cc:205] Tried to move back new op lower bound from 7333856986981392384 to 7333856986761555968. Current Snapshot: MvccSnapshot[applied={T|T < 7333856986981392384}]
I20260927 07:00:28.468007 13573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:28.468114 13573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:00:28.468863 13573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae1bb2c9119d400bbf53454b9fe23599: Sending DeleteTablet for 3 replicas of tablet d570753df2a44512a0f3d6391804e3de
I20260927 07:00:28.469200 13966 tablet_service.cc:1514] Processing DeleteTablet for tablet d570753df2a44512a0f3d6391804e3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:59964
I20260927 07:00:28.469283 13698 tablet_service.cc:1514] Processing DeleteTablet for tablet d570753df2a44512a0f3d6391804e3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:45934
I20260927 07:00:28.469410 13832 tablet_service.cc:1514] Processing DeleteTablet for tablet d570753df2a44512a0f3d6391804e3de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:00:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:45588
I20260927 07:00:28.469556 14069 tablet_replica.cc:330] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc: stopping tablet replica
I20260927 07:00:28.469681 14070 tablet_replica.cc:330] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: stopping tablet replica
I20260927 07:00:28.469691 14069 raft_consensus.cc:2229] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:28.469779 14070 raft_consensus.cc:2229] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:00:28.469776 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13635
I20260927 07:00:28.469871 14070 raft_consensus.cc:2258] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:28.469974 14069 raft_consensus.cc:2258] T d570753df2a44512a0f3d6391804e3de P 08eb48c6f28141c8b189b4e8e00f7ddc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:28.471304 14070 ts_tablet_manager.cc:1905] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:28.469614 14068 tablet_replica.cc:330] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: stopping tablet replica
I20260927 07:00:28.472585 14068 raft_consensus.cc:2229] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:00:28.472736 14068 raft_consensus.cc:2258] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:00:28.473196 14068 ts_tablet_manager.cc:1905] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:00:28.473351 14070 ts_tablet_manager.cc:1918] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:28.473430 14070 log.cc:1196] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/d570753df2a44512a0f3d6391804e3de
I20260927 07:00:28.473698 14070 ts_tablet_manager.cc:1939] T d570753df2a44512a0f3d6391804e3de P bd104521d3b64acf848c7bad7918e701: Deleting consensus metadata
I20260927 07:00:28.474211 13560 catalog_manager.cc:4841] TS bd104521d3b64acf848c7bad7918e701 (127.3.57.2:39207): tablet d570753df2a44512a0f3d6391804e3de (table test-table [id=c7e155ac7ad3496ba2929c3dcf7c456c]) successfully deleted
I20260927 07:00:28.475054 14068 ts_tablet_manager.cc:1918] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:00:28.475127 14068 log.cc:1196] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/d570753df2a44512a0f3d6391804e3de
I20260927 07:00:28.475387 14068 ts_tablet_manager.cc:1939] T d570753df2a44512a0f3d6391804e3de P 3aa727f601494e168d692bd3c4859262: Deleting consensus metadata
I20260927 07:00:28.475818 13559 catalog_manager.cc:4841] TS 3aa727f601494e168d692bd3c4859262 (127.3.57.3:45559): tablet d570753df2a44512a0f3d6391804e3de (table test-table [id=c7e155ac7ad3496ba2929c3dcf7c456c]) successfully deleted
W20260927 07:00:28.476600 13557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:40223 (error 108)
I20260927 07:00:28.476811 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13769
I20260927 07:00:28.477399 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:00:28.476920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40223 (local address 127.0.0.1:45938)
0927 07:00:28.477318 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 07:00:28.477360 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260927 07:00:28.477756 13557 catalog_manager.cc:4584] TS 08eb48c6f28141c8b189b4e8e00f7ddc (127.3.57.1:40223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d570753df2a44512a0f3d6391804e3de: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40223: connect: Connection refused (error 111)
I20260927 07:00:28.481616 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13903
I20260927 07:00:28.486174 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 13541
2026-09-27T07:00:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2113 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:00:28 dist-test-slave-mk3h krb5kdc[14074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:00:28 dist-test-slave-mk3h krb5kdc[14074](info): set up 2 sockets
Sep 27 07:00:28 dist-test-slave-mk3h krb5kdc[14074](info): commencing operation
krb5kdc: starting...
W20260927 07:00:29.507762 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:00:29 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:00:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangeradmin_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangeradmin_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangerlookup_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangerlookup_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
I20260927 07:00:29.665776 3300 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaZGoNp/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/postgres -l logfile start
2026-09-27 07:00:32.179 UTC [14104] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:00:32.179 UTC [14104] LOG: listening on IPv4 address "127.3.57.20", port 37381
2026-09-27 07:00:32.183 UTC [14104] LOG: listening on Unix socket "/tmp/.s.PGSQL.37381"
2026-09-27 07:00:32.205 UTC [14107] LOG: database system was shut down at 2026-09-27 07:00:30 UTC
2026-09-27 07:00:32.209 UTC [14104] LOG: database system is ready to accept connections
I20260927 07:00:33.168146 3300 mini_postgres.cc:96] Postgres bound to 37381
2026-09-27 07:00:33.174 UTC [14115] FATAL: database "slave" does not exist
127.3.57.20:37381 - accepting connections
I20260927 07:00:33.174856 3300 mini_ranger.cc:161] Starting Ranger...
I20260927 07:00:33.190836 3300 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:00:33.706624 3300 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:00:33.706758 3300 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin
2026-09-27 07:00:33,999 [I] DB FLAVOR :POSTGRES
2026-09-27 07:00:33,999 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:00:33,999 [I] Checking connection..
2026-09-27 07:00:33,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:00:34,255 [I] Checking connection passed.
2026-09-27 07:00:34,255 [I] --------- Verifying version history table ---------
2026-09-27 07:00:34,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:00:34,483 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:00:34,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:00:34,723 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:00:34,723 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:00:34,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:00:34.951 UTC [14179] WARNING: there is no transaction in progress
2026-09-27 07:00:34,957 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:00:34,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:00:35,222 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:00:35,222 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:00:35,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:00:35,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:00:35,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mk3h.c.gcp-upstream.internal','N') ;"
2026-09-27 07:00:35,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:00:36,243 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:00:36,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:00:36,493 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:00:36,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:00:36,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:00:36,994 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:00:36,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:00:37.975 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:37.991 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.007 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.024 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.051 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.073 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.129 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.136 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.151 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.161 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.172 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.186 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.196 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.202 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.208 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38.216 UTC [14287] WARNING: there is no transaction in progress
2026-09-27 07:00:38,843 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:00:38,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:00:39,104 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:00:39,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:00:39,361 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:00:39,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:00:39,598 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:00:39,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:00:39,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:00:40,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mk3h.c.gcp-upstream.internal';"
2026-09-27 07:00:40,307 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:00:40,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:00:40,548 [I] DB_PATCHES have already been applied
I20260927 07:00:40.553989 3300 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaZGoNp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:00:40.554054 3300 mini_ranger.cc:193] Using host: 127.3.57.20
I20260927 07:00:40.556943 3300 mini_ranger.cc:227] Ranger admin URL: http://127.3.57.20:37701
Sep 27, 2026 7:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:00:41.667071 3300 mini_ranger.cc:160] Time spent starting Ranger: real 8.492s user 0.000s sys 0.010s
Sep 27, 2026 7:01:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:01:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 27, 2026 7:01:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:01:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:01:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:01:03.110589 3300 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangerkms_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/rangerkms_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.57.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.57.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:01:03 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:01:03.196715 3300 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:01:03.205648 3300 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:01:04.026407 3300 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:01:04.026495 3300 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms
I20260927 07:01:04.026518 3300 mini_ranger_kms.cc:227] Using postgres at 127.3.57.20:37381
2026-09-27 07:01:04,201 [I] DB FLAVOR :POSTGRES
2026-09-27 07:01:04,201 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:01:04,201 [I] Checking connection
2026-09-27 07:01:04,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:01:04,429 [I] connection success
2026-09-27 07:01:04,429 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:01:04,429 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:01:04,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:01:04,685 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:01:04,685 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:01:04,685 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:01:04,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:01:04,935 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:01:05.350517 3300 mini_ranger_kms.cc:310] Created kms service
I20260927 07:01:05.532747 3300 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:01:05.575058 3300 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:01:05 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492463, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:05.954645 3300 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:01:05.954922 3300 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:01:05.954945 3300 mini_ranger_kms.cc:240] Using host: 127.3.57.20
I20260927 07:01:05.958628 3300 mini_ranger_kms.cc:276] Ranger KMS PID: 14467
I20260927 07:01:05.958768 3300 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.57.20:47773
Sep 27, 2026 7:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:01:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:01:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 07:01:08 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:08.248284 3300 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.052s user 0.000s sys 0.159s
I20260927 07:01:08.248389 3300 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:01:08.248471 3300 mini_ranger_kms.cc:401] 127.3.57.20:47773/kms/v1/keys
Sep 27 07:01:08 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
Sep 27 07:01:08 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492468, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
Sep 27 07:01:11 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:01:11.152032 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42059
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:42059
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:47773/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:11.232412 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:11.232609 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:11.232630 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:11.233793 14518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:01:11.233834 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:11.233847 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:11.233860 14518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:01:11.233872 14518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:01:11.235378 14518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.57.20:47773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:42059
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:11.235631 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:11.235889 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:11.241941 14523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:11.241947 14526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:11.241997 14518 server_base.cc:1034] running on GCE node
W20260927 07:01:11.241942 14524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:11.242343 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:11.242678 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:11.243850 14518 hybrid_clock.cc:648] HybridClock initialized: now 1790492471243842 us; error 36 us; skew 500 ppm
Sep 27 07:01:11 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:11.251327 14518 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:01:11.251781 14518 webserver.cc:460] Webserver started at http://127.3.57.62:40825/ using document root <none> and password file <none>
I20260927 07:01:11.251951 14518 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:11.251984 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:11.252066 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:11 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:11.296418 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "8488a34dea1e4e1ab511d7ec9a70bec5"
format_stamp: "Formatted at 2026-09-27 07:01:11 on dist-test-slave-mk3h"
server_key: "a93489d82885d9debab7dda5f4160bf3"
server_key_iv: "dfb411781179da1dc3f1209a900584de"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:11.296907 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "8488a34dea1e4e1ab511d7ec9a70bec5"
format_stamp: "Formatted at 2026-09-27 07:01:11 on dist-test-slave-mk3h"
server_key: "a93489d82885d9debab7dda5f4160bf3"
server_key_iv: "dfb411781179da1dc3f1209a900584de"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:11.298324 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:01:11.310082 14533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:11.310298 14518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:11.310415 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "8488a34dea1e4e1ab511d7ec9a70bec5"
format_stamp: "Formatted at 2026-09-27 07:01:11 on dist-test-slave-mk3h"
server_key: "a93489d82885d9debab7dda5f4160bf3"
server_key_iv: "dfb411781179da1dc3f1209a900584de"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:11.310508 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:11.321991 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:11.324031 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:11.324173 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:11.329713 14518 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:42059
I20260927 07:01:11.329789 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:42059 every 8 connection(s)
I20260927 07:01:11.330124 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:01:11.330792 14586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:11.333004 14586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: Bootstrap starting.
I20260927 07:01:11.333575 14586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:11.333850 14586 log.cc:824] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:11.334539 14586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: No bootstrap required, opened a new log
I20260927 07:01:11.336562 14586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } }
I20260927 07:01:11.336721 14586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:11.336798 14586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8488a34dea1e4e1ab511d7ec9a70bec5, State: Initialized, Role: FOLLOWER
I20260927 07:01:11.336978 14586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } }
I20260927 07:01:11.337092 14586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:01:11.337150 14586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:01:11.337190 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 14518
I20260927 07:01:11.337206 14586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:11.337283 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:01:11.337487 3300 external_mini_cluster.cc:1428] Setting key 831ea3f202aff3f4909df78fde3c21d9
I20260927 07:01:11.337908 14586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } }
I20260927 07:01:11.338048 14586 leader_election.cc:302] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8488a34dea1e4e1ab511d7ec9a70bec5; no voters:
I20260927 07:01:11.338222 14586 leader_election.cc:288] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:01:11.338267 14591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:01:11.338438 14591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [term 1 LEADER]: Becoming Leader. State: Replica: 8488a34dea1e4e1ab511d7ec9a70bec5, State: Running, Role: LEADER
I20260927 07:01:11.338531 14586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:01:11.338613 14591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } }
I20260927 07:01:11.339015 14591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8488a34dea1e4e1ab511d7ec9a70bec5. Latest consensus state: current_term: 1 leader_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } } }
I20260927 07:01:11.339149 14591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:11.339118 14592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8488a34dea1e4e1ab511d7ec9a70bec5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42059 } } }
I20260927 07:01:11.339283 14592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:11.340485 14600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:01:11.340682 14586 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:01:11.344323 14600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:01:11.350138 14600 catalog_manager.cc:1347] Generated new cluster ID: 696e19af3f02439198e8bb0349841691
I20260927 07:01:11.350224 14600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:01:11 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:11.354694 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:11.337978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.3.57.62:42059)
0927 07:01:11.338308 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:11.338317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:11.339752 (+ 1435us) server_negotiation.cc:407] Connection header received
0927 07:01:11.339896 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:11.339905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:11.339958 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:11.340087 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:11.340777 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:11.345322 (+ 4545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:11.346503 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:11.347192 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:11.347298 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:11.350796 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:11.350800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:11.350800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:11.350812 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:11.352773 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:11.353193 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:11.353196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:11.353196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:11.353295 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:11.353501 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:11.353502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:11.353503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:11.353826 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:11.353955 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:11.354095 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:01:11.354217 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":82,"threads_started":1}
I20260927 07:01:11.369283 14600 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:01:11.370096 14600 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:01:11.376185 14600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: Generated new TSK 0
I20260927 07:01:11.376425 14600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:01:12.071101 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:47773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:12.176149 14631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:12.176381 14631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:12.176421 14631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:12.177554 14631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:12.177624 14631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:12.177670 14631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:01:12.179158 14631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.57.20:47773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:12.179399 14631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:12.179702 14631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:12.185412 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.185425 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.185480 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:12.185822 14631 server_base.cc:1034] running on GCE node
I20260927 07:01:12.186035 14631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:12.186282 14631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:12.187449 14631 hybrid_clock.cc:648] HybridClock initialized: now 1790492472187440 us; error 34 us; skew 500 ppm
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:12.190136 14631 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:01:12.190670 14631 webserver.cc:460] Webserver started at http://127.3.57.1:43415/ using document root <none> and password file <none>
I20260927 07:01:12.190937 14631 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:12.191007 14631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:12.191129 14631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:12.205458 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "94fe263c3e9241a68abf84f1e345d0a2"
server_key_iv: "d64040710739fbd22df1a234cb0058a6"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.205858 14631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "94fe263c3e9241a68abf84f1e345d0a2"
server_key_iv: "d64040710739fbd22df1a234cb0058a6"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.207286 14631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:01:12.219853 14648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.220057 14631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:12.220161 14631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "94fe263c3e9241a68abf84f1e345d0a2"
server_key_iv: "d64040710739fbd22df1a234cb0058a6"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.220263 14631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:12.234846 14631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:12.236850 14631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:12.237007 14631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:12.237257 14631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:12.237653 14631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:12.237699 14631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.237726 14631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:12.237744 14631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.245723 14631 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:39711
I20260927 07:01:12.245803 14761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:39711 every 8 connection(s)
I20260927 07:01:12.246218 14631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:01:12.247483 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 14631
I20260927 07:01:12.247666 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:01:12.247880 3300 external_mini_cluster.cc:1428] Setting key bed40c1614b86b8ca095aedbc96ffa88
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:12.258571 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.246968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:51283 (local address 127.3.57.62:42059)
0927 07:01:12.247112 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.247114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.247329 (+ 215us) server_negotiation.cc:407] Connection header received
0927 07:01:12.248068 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.248069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.248085 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.248466 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:12.249254 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.250061 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.251541 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.251882 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.251958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.254634 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:12.254636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:12.254636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:12.254644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:12.256540 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.257024 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.257025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.257025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.257097 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.257387 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.257388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.257388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.257526 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:12.257596 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.258365 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.258454 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
I20260927 07:01:12.259318 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.246941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42059 (local address 127.3.57.1:51283)
0927 07:01:12.247178 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.247225 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.247946 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.248647 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.248651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.248895 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:12.249148 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.249153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.250175 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.250175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.251470 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.251472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.251985 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.251986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.252151 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.252755 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:12.252764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:12.254514 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:12.256638 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.256645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.256648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.256939 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.257166 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.257166 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.257167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.257294 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.257623 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:12.257630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:12.257802 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.258895 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.259065 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
I20260927 07:01:12.259912 14762 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42059
I20260927 07:01:12.260035 14762 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:12.260218 14762 heartbeater.cc:511] Master 127.3.57.62:42059 requested a full tablet report, sending...
I20260927 07:01:12.260818 14550 ts_manager.cc:194] Registered new tserver with Master: c70faf4ce8ae46d2bfb8f8d282bc73b3 (127.3.57.1:39711)
I20260927 07:01:12.261444 14550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:51283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:01:12.276175 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:47773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:12.383247 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:12.383478 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:12.383517 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:12.384701 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:12.384774 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:12.384852 14768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:01:12.386339 14768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.57.20:47773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:12.386689 14768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:12.386931 14768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:12.391806 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.392021 14775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.392140 14776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:12.392273 14768 server_base.cc:1034] running on GCE node
I20260927 07:01:12.392568 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:12.392850 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:12.393990 14768 hybrid_clock.cc:648] HybridClock initialized: now 1790492472393972 us; error 36 us; skew 500 ppm
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:12.396643 14768 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:01:12.397110 14768 webserver.cc:460] Webserver started at http://127.3.57.2:36055/ using document root <none> and password file <none>
I20260927 07:01:12.397338 14768 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:12.397406 14768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:12.397536 14768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:12.413089 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "b686ac7a80dc41c1b2a627a4cc65148c"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "de655b0731a53dd7e1fcb003feb8491c"
server_key_iv: "49864c4e00675a129b216f369c1c59be"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.414784 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "b686ac7a80dc41c1b2a627a4cc65148c"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "de655b0731a53dd7e1fcb003feb8491c"
server_key_iv: "49864c4e00675a129b216f369c1c59be"
server_key_version: "kuduclusterkey@0"
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:12.417783 14768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:01:12.434432 14785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.434707 14768 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260927 07:01:12.434866 14768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "b686ac7a80dc41c1b2a627a4cc65148c"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "de655b0731a53dd7e1fcb003feb8491c"
server_key_iv: "49864c4e00675a129b216f369c1c59be"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.435048 14768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:12.466437 14768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:12.468914 14768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:12.469103 14768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:12.469354 14768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:12.469794 14768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:12.469885 14768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.469957 14768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:12.470013 14768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.476627 14768 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:35283
I20260927 07:01:12.477092 14768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:01:12.477700 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:35283 every 8 connection(s)
I20260927 07:01:12.483773 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 14768
I20260927 07:01:12.483857 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:01:12.484041 3300 external_mini_cluster.cc:1428] Setting key f44f712d1b8f17fdcbd69a29d4926336
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:12.490624 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.478778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:44607 (local address 127.3.57.62:42059)
0927 07:01:12.478823 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.478825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.478836 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:01:12.479421 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.479422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.479434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.479505 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:12.480202 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.480972 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.482386 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.482705 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.482785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.487450 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:12.487452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:12.487452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:12.487460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:12.489149 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.489612 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.489613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.489614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.489668 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.489931 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.489932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.489932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.490054 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:12.490093 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.490440 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.490529 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:01:12.491120 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.478479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42059 (local address 127.3.57.2:44607)
0927 07:01:12.478671 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.478713 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.479304 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.479610 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.479613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.479840 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:12.480101 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.480106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.481065 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.481065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.482303 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.482305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.482797 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.482798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.483377 (+ 579us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.484285 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:12.484295 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:12.487354 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:12.489240 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.489246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.489249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.489526 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.489745 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.489746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.489746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.489861 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.490135 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:12.490140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:12.490300 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.490755 (+ 455us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.490905 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260927 07:01:12.491648 14899 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42059
I20260927 07:01:12.491745 14899 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:12.491925 14899 heartbeater.cc:511] Master 127.3.57.62:42059 requested a full tablet report, sending...
I20260927 07:01:12.492280 14550 ts_manager.cc:194] Registered new tserver with Master: b686ac7a80dc41c1b2a627a4cc65148c (127.3.57.2:35283)
I20260927 07:01:12.492738 14550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:44607
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:01:12.512893 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:47773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:12.632870 14904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:12.633134 14904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:12.633167 14904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:12.634811 14904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:12.634860 14904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:12.634897 14904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:01:12.636924 14904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.57.20:47773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:42059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:12.637141 14904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:12.637413 14904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:12.643615 14910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.643712 14913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:12.643851 14911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:12.644944 14904 server_base.cc:1034] running on GCE node
I20260927 07:01:12.645162 14904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:12.645437 14904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:12.646615 14904 hybrid_clock.cc:648] HybridClock initialized: now 1790492472646584 us; error 43 us; skew 500 ppm
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:12.649818 14904 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:01:12.650264 14904 webserver.cc:460] Webserver started at http://127.3.57.3:33121/ using document root <none> and password file <none>
I20260927 07:01:12.650487 14904 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:12.650534 14904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:12.650658 14904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:12.664031 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "8351cb634c0c48dd956160e6fa4e7566"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "374fb50f47d63c44724e8b49de63c5b9"
server_key_iv: "27b39415d1647889a9a6722ba640c39c"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.664470 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "8351cb634c0c48dd956160e6fa4e7566"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "374fb50f47d63c44724e8b49de63c5b9"
server_key_iv: "27b39415d1647889a9a6722ba640c39c"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.665892 14904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:01:12.685740 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.686386 14904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:01:12.686474 14904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "8351cb634c0c48dd956160e6fa4e7566"
format_stamp: "Formatted at 2026-09-27 07:01:12 on dist-test-slave-mk3h"
server_key: "374fb50f47d63c44724e8b49de63c5b9"
server_key_iv: "27b39415d1647889a9a6722ba640c39c"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:12.686568 14904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:12.707127 14904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:12.708976 14904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:12.709125 14904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:12.709357 14904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:12.709744 14904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:12.709796 14904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.709838 14904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:12.709864 14904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:12.717036 14904 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33887
I20260927 07:01:12.717139 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33887 every 8 connection(s)
I20260927 07:01:12.717453 14904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:01:12.723134 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 14904
I20260927 07:01:12.723654 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:01:12.723881 3300 external_mini_cluster.cc:1428] Setting key 1d659f256dfc166e5864a163f449ef93
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:12.735749 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.718216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:33507 (local address 127.3.57.62:42059)
0927 07:01:12.718294 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.718297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.718550 (+ 253us) server_negotiation.cc:407] Connection header received
0927 07:01:12.719854 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.719856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.719869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.719921 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:12.720675 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.721172 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.722604 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.722811 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.722892 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.725331 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:12.725332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:12.725333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:12.725340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:12.726926 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.727424 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.727426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.727426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.727482 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.727793 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.727794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.727795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.727917 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:12.727957 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.735568 (+ 7611us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.735657 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:01:12.735944 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.718181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:42059 (local address 127.3.57.3:33507)
0927 07:01:12.718387 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.718441 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.719571 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.720057 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.720063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.720283 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:12.720563 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.720569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.721296 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.721297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.722497 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.722500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.722897 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.722898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.723039 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.723625 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:12.723636 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:12.725220 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:12.727034 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.727039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.727042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.727332 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.727584 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:12.727584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:12.727585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:12.727710 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:12.731819 (+ 4109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:12.731825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:12.731991 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.735511 (+ 3520us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.735679 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260927 07:01:12.736407 15034 heartbeater.cc:347] Connected to a master server at 127.3.57.62:42059
I20260927 07:01:12.736515 15034 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:12.736697 15034 heartbeater.cc:511] Master 127.3.57.62:42059 requested a full tablet report, sending...
I20260927 07:01:12.737157 14550 ts_manager.cc:194] Registered new tserver with Master: 8351cb634c0c48dd956160e6fa4e7566 (127.3.57.3:33887)
I20260927 07:01:12.737560 14550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:33507
I20260927 07:01:12.737686 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:01:12.745599 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.739009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.3.57.62:42059)
0927 07:01:12.739042 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.739044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.739280 (+ 236us) server_negotiation.cc:407] Connection header received
0927 07:01:12.739381 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.739382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.739391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.739419 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:12.740260 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.740854 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.741588 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.741786 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.741859 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.743007 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:12.743008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:12.743009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:12.743022 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:12.744502 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.744859 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.744860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.744861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.744912 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:12.745114 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:12.745115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:12.745115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:12.745221 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:12.745326 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.745428 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.745499 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 07:01:12.747356 14550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:01:12.749307 14550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:01:12.759941 15049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.754831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39711 (local address 127.0.0.1:50602)
0927 07:01:12.755885 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.755894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.755942 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.756201 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.756202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.756204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:12.756243 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.756244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.757480 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.757481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.758541 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.758544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.759628 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.759629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.759715 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.759740 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.759812 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.759862 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":54,"threads_started":1}
I20260927 07:01:12.760248 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.754998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.3.57.1:39711)
0927 07:01:12.755834 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.755839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.755997 (+ 158us) server_negotiation.cc:407] Connection header received
0927 07:01:12.756041 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.756043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.756059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.756126 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:12.756318 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.757381 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.758661 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.759425 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.759496 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.759592 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.760144 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.760185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":56,"threads_started":1}
I20260927 07:01:12.760288 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.751496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:35283 (local address 127.0.0.1:49814)
0927 07:01:12.752164 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.752201 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.752261 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.752706 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.752709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.752711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:12.752770 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.752772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.754042 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.754043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.755258 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.755261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.759846 (+ 4585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.759847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.759919 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.759923 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.759977 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.760211 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":3231,"threads_started":1}
I20260927 07:01:12.760911 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.751497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33887 (local address 127.0.0.1:35700)
0927 07:01:12.755636 (+ 4139us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.755648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.755711 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.756536 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.756537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.756539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:12.756583 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.756585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.757801 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.757802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.758905 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.758907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.760689 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.760690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.760757 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.760761 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.760808 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.760843 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4121,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260927 07:01:12.761122 14696 tablet_service.cc:1467] Processing CreateTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e (DEFAULT_TABLE table=test-table [id=448f6d5c5dae4ba5898defe13378ad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:12.761334 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.751987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.3.57.3:33887)
0927 07:01:12.752668 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.752672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.756330 (+ 3658us) server_negotiation.cc:407] Connection header received
0927 07:01:12.756370 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.756372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.756387 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.756459 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:12.756657 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.757703 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.759006 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.760501 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.760573 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.760638 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.761215 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.761257 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":63,"threads_started":1}
I20260927 07:01:12.761505 14696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da5b2194ff4a708e6afb86dfa4205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:12.761889 14949 tablet_service.cc:1467] Processing CreateTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e (DEFAULT_TABLE table=test-table [id=448f6d5c5dae4ba5898defe13378ad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:12.762172 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da5b2194ff4a708e6afb86dfa4205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:12.765230 15052 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: Bootstrap starting.
I20260927 07:01:12.765424 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.751835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.3.57.2:35283)
0927 07:01:12.752402 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.752407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.752434 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:01:12.752468 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.752470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.752484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.752562 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:12.752846 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.753941 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.755372 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.756158 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.756230 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.756302 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.765287 (+ 8985us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.765336 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":82,"threads_started":1}
I20260927 07:01:12.765944 15052 tablet_bootstrap.cc:654] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:12.766273 15052 log.cc:824] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:12.766287 14822 tablet_service.cc:1467] Processing CreateTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e (DEFAULT_TABLE table=test-table [id=448f6d5c5dae4ba5898defe13378ad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:12.766476 15053 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Bootstrap starting.
I20260927 07:01:12.766592 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2da5b2194ff4a708e6afb86dfa4205e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:12.767625 15053 tablet_bootstrap.cc:654] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:12.767953 15053 log.cc:824] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:12.768697 15052 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: No bootstrap required, opened a new log
I20260927 07:01:12.768805 15052 ts_tablet_manager.cc:1397] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 07:01:12.771438 15052 raft_consensus.cc:348] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.771627 15052 raft_consensus.cc:374] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:12.771703 15052 raft_consensus.cc:729] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70faf4ce8ae46d2bfb8f8d282bc73b3, State: Initialized, Role: FOLLOWER
I20260927 07:01:12.771842 15052 consensus_queue.cc:260] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.772101 15052 ts_tablet_manager.cc:1428] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:01:12.772256 14762 heartbeater.cc:503] Master 127.3.57.62:42059 was elected leader, sending a full tablet report...
I20260927 07:01:12.772768 15053 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: No bootstrap required, opened a new log
I20260927 07:01:12.772825 15053 ts_tablet_manager.cc:1397] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 07:01:12.775125 15053 raft_consensus.cc:348] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.775279 15053 raft_consensus.cc:374] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:12.775313 15053 raft_consensus.cc:729] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8351cb634c0c48dd956160e6fa4e7566, State: Initialized, Role: FOLLOWER
I20260927 07:01:12.775470 15053 consensus_queue.cc:260] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.775826 15053 ts_tablet_manager.cc:1428] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 07:01:12.775946 15034 heartbeater.cc:503] Master 127.3.57.62:42059 was elected leader, sending a full tablet report...
I20260927 07:01:12.776474 15056 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Bootstrap starting.
I20260927 07:01:12.777444 15056 tablet_bootstrap.cc:654] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:12.777750 15056 log.cc:824] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:12.778502 15056 tablet_bootstrap.cc:492] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: No bootstrap required, opened a new log
I20260927 07:01:12.778592 15056 ts_tablet_manager.cc:1397] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:01:12.780551 15056 raft_consensus.cc:348] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.780778 15056 raft_consensus.cc:374] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:12.780840 15056 raft_consensus.cc:729] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b686ac7a80dc41c1b2a627a4cc65148c, State: Initialized, Role: FOLLOWER
I20260927 07:01:12.780948 15056 consensus_queue.cc:260] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.781186 15056 ts_tablet_manager.cc:1428] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:01:12.781297 14899 heartbeater.cc:503] Master 127.3.57.62:42059 was elected leader, sending a full tablet report...
Sep 27 07:01:12 dist-test-slave-mk3h krb5kdc[14074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492472, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:12.872449 15060 raft_consensus.cc:482] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:01:12.872577 15060 raft_consensus.cc:504] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.872951 15060 leader_election.cc:288] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70faf4ce8ae46d2bfb8f8d282bc73b3 (127.3.57.1:39711), 8351cb634c0c48dd956160e6fa4e7566 (127.3.57.3:33887)
I20260927 07:01:12.876919 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.873133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33887 (local address 127.3.57.2:33247)
0927 07:01:12.873237 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.873248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.873289 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.873650 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.873652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.873654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:12.873724 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.873726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.874680 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.874681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.875904 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.875907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.876676 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.876678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.876755 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.876758 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.876810 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.876849 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 07:01:12.877601 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.873372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:33247 (local address 127.3.57.3:33887)
0927 07:01:12.873431 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.873433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.873444 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:01:12.873474 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.873475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.873488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.873592 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:12.873799 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.874561 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.876009 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.876561 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.876632 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.876643 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.877489 (+ 846us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.877532 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:01:12.877851 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da5b2194ff4a708e6afb86dfa4205e" candidate_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8351cb634c0c48dd956160e6fa4e7566" is_pre_election: true
I20260927 07:01:12.878000 14988 raft_consensus.cc:2454] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b686ac7a80dc41c1b2a627a4cc65148c in term 0.
I20260927 07:01:12.878335 14787 leader_election.cc:302] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8351cb634c0c48dd956160e6fa4e7566, b686ac7a80dc41c1b2a627a4cc65148c; no voters:
I20260927 07:01:12.878656 15060 raft_consensus.cc:2781] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:01:12.878714 15060 raft_consensus.cc:482] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:01:12.878738 15060 raft_consensus.cc:3037] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:12.879649 15060 raft_consensus.cc:504] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.879774 15060 leader_election.cc:288] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [CANDIDATE]: Term 1 election: Requested vote from peers c70faf4ce8ae46d2bfb8f8d282bc73b3 (127.3.57.1:39711), 8351cb634c0c48dd956160e6fa4e7566 (127.3.57.3:33887)
I20260927 07:01:12.880326 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da5b2194ff4a708e6afb86dfa4205e" candidate_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8351cb634c0c48dd956160e6fa4e7566"
I20260927 07:01:12.880403 14988 raft_consensus.cc:3037] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:12.881248 14988 raft_consensus.cc:2454] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b686ac7a80dc41c1b2a627a4cc65148c in term 1.
I20260927 07:01:12.881404 14787 leader_election.cc:302] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8351cb634c0c48dd956160e6fa4e7566, b686ac7a80dc41c1b2a627a4cc65148c; no voters:
I20260927 07:01:12.881475 15060 raft_consensus.cc:2781] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:01:12.881596 15060 raft_consensus.cc:686] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 1 LEADER]: Becoming Leader. State: Replica: b686ac7a80dc41c1b2a627a4cc65148c, State: Running, Role: LEADER
I20260927 07:01:12.881672 15060 consensus_queue.cc:237] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } }
I20260927 07:01:12.882436 14550 catalog_manager.cc:5495] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c reported cstate change: term changed from 0 to 1, leader changed from <none> to b686ac7a80dc41c1b2a627a4cc65148c (127.3.57.2). New cstate: current_term: 1 leader_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20260927 07:01:12.889398 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.873646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39711 (local address 127.3.57.2:35345)
0927 07:01:12.876982 (+ 3336us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.876992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:12.877029 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:12.884792 (+ 7763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:12.884793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:12.884795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:12.884828 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.884829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.887451 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.887452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.888445 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:12.888448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.889201 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.889202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:12.889257 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.889261 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:01:12.889302 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 07:01:12.889337 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3325,"thread_start_us":84,"threads_started":1}
I20260927 07:01:12.889649 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.873827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:35345 (local address 127.3.57.1:39711)
0927 07:01:12.873867 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.873869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.877084 (+ 3215us) server_negotiation.cc:407] Connection header received
0927 07:01:12.877111 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.877112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.877126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.877169 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:12.884919 (+ 7750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.885600 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.888544 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.889086 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.889157 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.889169 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.889529 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.889579 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:01:12.889943 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da5b2194ff4a708e6afb86dfa4205e" candidate_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" is_pre_election: true
I20260927 07:01:12.889964 14715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2da5b2194ff4a708e6afb86dfa4205e" candidate_uuid: "b686ac7a80dc41c1b2a627a4cc65148c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3"
I20260927 07:01:12.890092 14716 raft_consensus.cc:2454] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b686ac7a80dc41c1b2a627a4cc65148c in term 0.
I20260927 07:01:12.901886 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.898667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.3.57.2:35283)
0927 07:01:12.898702 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:12.898703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:12.898710 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:01:12.898737 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:12.898738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:12.898755 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:12.898836 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:01:12.899055 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.899769 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.900922 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:12.901243 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:12.901310 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:12.901494 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:01:12.901607 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:01:12.901685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:12.901776 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 07:01:12.901798 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:01:12.904075 14715 raft_consensus.cc:3037] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:12.904093 14988 raft_consensus.cc:1261] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 1 FOLLOWER]: Refusing update from remote peer b686ac7a80dc41c1b2a627a4cc65148c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:01:12.904420 15062 consensus_queue.cc:1035] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8351cb634c0c48dd956160e6fa4e7566" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:01:12.904951 14715 raft_consensus.cc:1261] T e2da5b2194ff4a708e6afb86dfa4205e P c70faf4ce8ae46d2bfb8f8d282bc73b3 [term 1 FOLLOWER]: Refusing update from remote peer b686ac7a80dc41c1b2a627a4cc65148c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:01:12.905309 15062 consensus_queue.cc:1035] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70faf4ce8ae46d2bfb8f8d282bc73b3" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 39711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:01:12.908027 15065 mvcc.cc:205] Tried to move back new op lower bound from 7333857169012559872 to 7333857168923742208. Current Snapshot: MvccSnapshot[applied={T|T < 7333857169012559872}]
I20260927 07:01:12.908816 15068 mvcc.cc:205] Tried to move back new op lower bound from 7333857169012559872 to 7333857168923742208. Current Snapshot: MvccSnapshot[applied={T|T < 7333857169012559872}]
I20260927 07:01:12.913430 14549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:01:12.913527 14549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:01:12.914273 14549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8488a34dea1e4e1ab511d7ec9a70bec5: Sending DeleteTablet for 3 replicas of tablet e2da5b2194ff4a708e6afb86dfa4205e
I20260927 07:01:12.914647 14696 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:50602
I20260927 07:01:12.914779 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:49814
I20260927 07:01:12.914965 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet e2da5b2194ff4a708e6afb86dfa4205e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:12 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35700
I20260927 07:01:12.915462 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 14631
I20260927 07:01:12.915689 15066 mvcc.cc:205] Tried to move back new op lower bound from 7333857169012559872 to 7333857168923742208. Current Snapshot: MvccSnapshot[applied={T|T < 7333857169012559872}]
I20260927 07:01:12.923312 15074 tablet_replica.cc:330] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: stopping tablet replica
I20260927 07:01:12.923424 15074 raft_consensus.cc:2229] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:01:12.923509 15074 raft_consensus.cc:2258] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:01:12.923739 15075 tablet_replica.cc:330] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: stopping tablet replica
I20260927 07:01:12.923841 15075 raft_consensus.cc:2229] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:01:12.923940 15075 raft_consensus.cc:2258] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:01:12.924361 15075 ts_tablet_manager.cc:1905] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:01:12.924496 15074 ts_tablet_manager.cc:1905] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:01:12.926074 15075 ts_tablet_manager.cc:1918] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:01:12.926191 15075 log.cc:1196] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/e2da5b2194ff4a708e6afb86dfa4205e
I20260927 07:01:12.926815 15075 ts_tablet_manager.cc:1939] T e2da5b2194ff4a708e6afb86dfa4205e P 8351cb634c0c48dd956160e6fa4e7566: Deleting consensus metadata
I20260927 07:01:12.927264 14535 catalog_manager.cc:4841] TS 8351cb634c0c48dd956160e6fa4e7566 (127.3.57.3:33887): tablet e2da5b2194ff4a708e6afb86dfa4205e (table test-table [id=448f6d5c5dae4ba5898defe13378ad82]) successfully deleted
I20260927 07:01:12.934020 15074 ts_tablet_manager.cc:1918] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:01:12.934170 15074 log.cc:1196] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/e2da5b2194ff4a708e6afb86dfa4205e
I20260927 07:01:12.934494 15074 ts_tablet_manager.cc:1939] T e2da5b2194ff4a708e6afb86dfa4205e P b686ac7a80dc41c1b2a627a4cc65148c: Deleting consensus metadata
W20260927 07:01:12.934969 14537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:39711 (error 108)
I20260927 07:01:12.935213 14537 catalog_manager.cc:4841] TS b686ac7a80dc41c1b2a627a4cc65148c (127.3.57.2:35283): tablet e2da5b2194ff4a708e6afb86dfa4205e (table test-table [id=448f6d5c5dae4ba5898defe13378ad82]) successfully deleted
I20260927 07:01:12.935261 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 14768
I20260927 07:01:12.936483 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:12.936273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:39711 (local address 127.0.0.1:50610)
0927 07:01:12.936408 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:12.936444 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:39711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260927 07:01:12.936919 14537 catalog_manager.cc:4584] TS c70faf4ce8ae46d2bfb8f8d282bc73b3 (127.3.57.1:39711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2da5b2194ff4a708e6afb86dfa4205e: Network error: Client connection negotiation failed: client connection to 127.3.57.1:39711: connect: Connection refused (error 111)
I20260927 07:01:12.941025 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 14904
I20260927 07:01:12.947778 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 14518
I20260927 07:01:12.953867 3300 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:01:13.287698 3300 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:01:13.287784 3300 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:01:13.631668 3300 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:01:13.631 UTC [14104] LOG: received smart shutdown request
2026-09-27 07:01:13.634 UTC [14104] LOG: background worker "logical replication launcher" (PID 14113) exited with exit code 1
2026-09-27 07:01:13.635 UTC [14108] LOG: shutting down
2026-09-27 07:01:13.778 UTC [14104] LOG: database system is shut down
2026-09-27T07:01:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45398 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:01:13 dist-test-slave-mk3h krb5kdc[15086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:01:13 dist-test-slave-mk3h krb5kdc[15086](info): set up 2 sockets
Sep 27 07:01:13 dist-test-slave-mk3h krb5kdc[15086](info): commencing operation
krb5kdc: starting...
W20260927 07:01:14.915623 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:01:14 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:01:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangeradmin_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangeradmin_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangerlookup_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangerlookup_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
I20260927 07:01:15.083173 3300 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskaZGoNp/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/postgres -l logfile start
2026-09-27 07:01:17.595 UTC [15116] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:01:17.595 UTC [15116] LOG: listening on IPv4 address "127.3.57.20", port 42405
2026-09-27 07:01:17.598 UTC [15116] LOG: listening on Unix socket "/tmp/.s.PGSQL.42405"
2026-09-27 07:01:17.635 UTC [15119] LOG: database system was shut down at 2026-09-27 07:01:16 UTC
2026-09-27 07:01:17.639 UTC [15116] LOG: database system is ready to accept connections
I20260927 07:01:18.599365 3300 mini_postgres.cc:96] Postgres bound to 42405
2026-09-27 07:01:18.605 UTC [15127] FATAL: database "slave" does not exist
127.3.57.20:42405 - accepting connections
I20260927 07:01:18.605872 3300 mini_ranger.cc:161] Starting Ranger...
I20260927 07:01:18.621219 3300 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:01:19.124076 3300 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:01:19.124178 3300 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin
2026-09-27 07:01:19,404 [I] DB FLAVOR :POSTGRES
2026-09-27 07:01:19,405 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:01:19,405 [I] Checking connection..
2026-09-27 07:01:19,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:01:19,659 [I] Checking connection passed.
2026-09-27 07:01:19,659 [I] --------- Verifying version history table ---------
2026-09-27 07:01:19,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:01:19,932 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:01:19,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:01:20,209 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:01:20,209 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:01:20,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:01:20.443 UTC [15191] WARNING: there is no transaction in progress
2026-09-27 07:01:20,450 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:01:20,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:01:20,699 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:01:20,700 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:01:20,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:01:20,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:01:21,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mk3h.c.gcp-upstream.internal','N') ;"
2026-09-27 07:01:21,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:01:21,677 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:01:21,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:01:21,935 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:01:21,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:01:22,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:01:22,416 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:01:22,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:01:23.386 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.403 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.420 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.436 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.453 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.474 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.529 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.537 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.551 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.561 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.571 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.584 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.593 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.600 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.606 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:23.613 UTC [15299] WARNING: there is no transaction in progress
2026-09-27 07:01:24,243 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:01:24,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:01:24,485 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:01:24,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:01:24,724 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:01:24,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:01:24,974 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:01:24,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:01:25,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:01:25,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mk3h.c.gcp-upstream.internal';"
2026-09-27 07:01:25,674 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:01:25,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:01:25,897 [I] DB_PATCHES have already been applied
I20260927 07:01:25.903084 3300 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskaZGoNp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:01:25.903157 3300 mini_ranger.cc:193] Using host: 127.3.57.20
I20260927 07:01:25.906279 3300 mini_ranger.cc:227] Ranger admin URL: http://127.3.57.20:48969
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:01:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:01:27.034999 3300 mini_ranger.cc:160] Time spent starting Ranger: real 8.429s user 0.001s sys 0.008s
Sep 27, 2026 7:01:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:01:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 7:01:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:01:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:01:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:01:47.336551 3300 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.57.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.57.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangerkms_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.57.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/rangerkms_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.57.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.57.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.57.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/HTTP_127.3.57.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:01:47 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:01:47.415740 3300 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:01:47.424131 3300 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:01:48.237512 3300 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:01:48.237576 3300 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms
I20260927 07:01:48.237588 3300 mini_ranger_kms.cc:227] Using postgres at 127.3.57.20:42405
2026-09-27 07:01:48,398 [I] DB FLAVOR :POSTGRES
2026-09-27 07:01:48,398 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:01:48,398 [I] Checking connection
2026-09-27 07:01:48,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:01:48,643 [I] connection success
2026-09-27 07:01:48,643 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:01:48,644 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:01:48,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:01:48,895 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:01:48,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:01:48,895 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:01:48,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.57.20:42405/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:01:49,171 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:01:49.613200 3300 mini_ranger_kms.cc:310] Created kms service
I20260927 07:01:49.827689 3300 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:01:49.871460 3300 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:01:49 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:50.294625 3300 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:01:50.294734 3300 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms:/tmp/dist-test-taskaZGoNp/build/release/bin/postgresql.jar:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskaZGoNp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:01:50.294752 3300 mini_ranger_kms.cc:240] Using host: 127.3.57.20
I20260927 07:01:50.298714 3300 mini_ranger_kms.cc:276] Ranger KMS PID: 15479
I20260927 07:01:50.298795 3300 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.57.20:48247
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:01:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 07:01:52 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:52.608001 3300 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.192s user 0.002s sys 0.142s
I20260927 07:01:52.608107 3300 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:01:52.608157 3300 mini_ranger_kms.cc:401] 127.3.57.20:48247/kms/v1/keys
Sep 27 07:01:52 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
Sep 27 07:01:53 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.57.20@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
Sep 27 07:01:55 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:01:55.614671 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44421
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44421
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:55.694160 15530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:55.694348 15530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:55.694370 15530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:55.695493 15530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:01:55.695531 15530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:55.695561 15530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:55.695598 15530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:01:55.695624 15530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:01:55.697091 15530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.57.20:48247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44421
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:55.697324 15530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:55.697566 15530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:55.703575 15538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:55.703613 15530 server_base.cc:1034] running on GCE node
W20260927 07:01:55.703586 15536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:55.703717 15535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:55.704226 15530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:55.704510 15530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:55.705670 15530 hybrid_clock.cc:648] HybridClock initialized: now 1790492515705658 us; error 30 us; skew 500 ppm
Sep 27 07:01:55 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:55.712596 15530 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:01:55.713068 15530 webserver.cc:460] Webserver started at http://127.3.57.62:35339/ using document root <none> and password file <none>
I20260927 07:01:55.713305 15530 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:55.713344 15530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:55.713482 15530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:55 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:55.761030 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "71a65ffebfcd4d63a5f5af97301d2394"
format_stamp: "Formatted at 2026-09-27 07:01:55 on dist-test-slave-mk3h"
server_key: "90194fddcc7065b541a25f6a7c6e956c"
server_key_iv: "57eb28c0621cab6487676aaa181d1bfb"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:55.761432 15530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "71a65ffebfcd4d63a5f5af97301d2394"
format_stamp: "Formatted at 2026-09-27 07:01:55 on dist-test-slave-mk3h"
server_key: "90194fddcc7065b541a25f6a7c6e956c"
server_key_iv: "57eb28c0621cab6487676aaa181d1bfb"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:55.762887 15530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:01:55.773865 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:55.774111 15530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:01:55.774266 15530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "71a65ffebfcd4d63a5f5af97301d2394"
format_stamp: "Formatted at 2026-09-27 07:01:55 on dist-test-slave-mk3h"
server_key: "90194fddcc7065b541a25f6a7c6e956c"
server_key_iv: "57eb28c0621cab6487676aaa181d1bfb"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:55.774355 15530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:55.793200 15530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:55.795233 15530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:55.795377 15530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:55.800362 15530 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:44421
I20260927 07:01:55.800508 15597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:44421 every 8 connection(s)
I20260927 07:01:55.800788 15530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:01:55.801456 15598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:55.803730 15598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Bootstrap starting.
I20260927 07:01:55.804303 15598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:55.804610 15598 log.cc:824] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:55.805258 15598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: No bootstrap required, opened a new log
I20260927 07:01:55.807219 15598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:55.807356 15598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:55.807379 15598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a65ffebfcd4d63a5f5af97301d2394, State: Initialized, Role: FOLLOWER
I20260927 07:01:55.807528 15598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:55.807626 15598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:01:55.807664 15598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:01:55.807710 15598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:55.808300 15598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:55.808379 15598 leader_election.cc:302] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a65ffebfcd4d63a5f5af97301d2394; no voters:
I20260927 07:01:55.808547 15598 leader_election.cc:288] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:01:55.808568 15601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:01:55.808676 15601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 LEADER]: Becoming Leader. State: Replica: 71a65ffebfcd4d63a5f5af97301d2394, State: Running, Role: LEADER
I20260927 07:01:55.808751 15601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:55.808784 15598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:01:55.809041 15601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a65ffebfcd4d63a5f5af97301d2394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } } }
I20260927 07:01:55.809221 15601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:55.809401 15601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a65ffebfcd4d63a5f5af97301d2394. Latest consensus state: current_term: 1 leader_uuid: "71a65ffebfcd4d63a5f5af97301d2394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } } }
I20260927 07:01:55.809484 15601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:55.809794 15608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:01:55.810307 15608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:01:55.810602 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 15530
I20260927 07:01:55.810691 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:01:55.810881 3300 external_mini_cluster.cc:1428] Setting key ba3365f7e65a4f9f6b8875405644bf46
I20260927 07:01:55.811619 15598 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:01:55.816162 15608 catalog_manager.cc:1347] Generated new cluster ID: 0923a7b316514d209af27905e6b86c53
I20260927 07:01:55.816293 15608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:01:55 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:55.825937 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:55.811473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.3.57.62:44421)
0927 07:01:55.813995 (+ 2522us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:55.814013 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:55.814050 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:01:55.814266 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:55.814271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:55.814333 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:55.814503 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:55.815332 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:55.817291 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:55.818289 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:55.818974 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:55.819073 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:55.821763 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:55.821767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:55.821767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:55.821777 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:55.823975 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:55.824344 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:55.824346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:55.824347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:55.824447 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:55.824623 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:55.824624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:55.824624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:55.824930 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:55.825473 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:55.825539 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:01:55.825662 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2425,"thread_start_us":73,"threads_started":1}
I20260927 07:01:55.829843 15608 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:01:55.830399 15608 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:01:55.836838 15608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Generated new TSK 0
I20260927 07:01:55.837056 15608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:01:56.555029 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:56.672755 15643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:56.673028 15643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:56.673069 15643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:56.674779 15643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:56.674839 15643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:56.674878 15643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:01:56.677001 15643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:56.677249 15643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:56.677507 15643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:56.683468 15648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:56.683748 15643 server_base.cc:1034] running on GCE node
W20260927 07:01:56.683835 15651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:56.683934 15649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:56.684127 15643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:56.684383 15643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:56.685600 15643 hybrid_clock.cc:648] HybridClock initialized: now 1790492516685580 us; error 33 us; skew 500 ppm
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:56.689168 15643 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:01:56.689734 15643 webserver.cc:460] Webserver started at http://127.3.57.1:35761/ using document root <none> and password file <none>
I20260927 07:01:56.690047 15643 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:56.690110 15643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:56.690287 15643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:56.706277 15643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "75e55c458b24420b83d4f32738af095d"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "0b3b7d787fdb5107734326af4af8ca38"
server_key_iv: "c4e9ddf06d1d9355843fb584da5be044"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.706705 15643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "75e55c458b24420b83d4f32738af095d"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "0b3b7d787fdb5107734326af4af8ca38"
server_key_iv: "c4e9ddf06d1d9355843fb584da5be044"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.709434 15643 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 07:01:56.722582 15658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:56.722867 15643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:56.723057 15643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "75e55c458b24420b83d4f32738af095d"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "0b3b7d787fdb5107734326af4af8ca38"
server_key_iv: "c4e9ddf06d1d9355843fb584da5be044"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.723228 15643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:56.757071 15643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:56.759481 15643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:56.759696 15643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:56.759966 15643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:56.760355 15643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:56.760408 15643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:56.760440 15643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:56.760469 15643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:56.769138 15643 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:42689
I20260927 07:01:56.769614 15643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:01:56.770287 15773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:42689 every 8 connection(s)
I20260927 07:01:56.772267 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 15643
I20260927 07:01:56.772357 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:01:56.772552 3300 external_mini_cluster.cc:1428] Setting key 2111575255f17b2d59690c8560d2e012
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:56.781263 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:56.770934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:45087 (local address 127.3.57.62:44421)
0927 07:01:56.771095 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:56.771097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:56.771299 (+ 202us) server_negotiation.cc:407] Connection header received
0927 07:01:56.771891 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:56.771892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:56.771907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:56.771978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:56.772770 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:56.773480 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:56.774951 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:56.775266 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:56.775328 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:56.777651 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:56.777653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:56.777653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:56.777661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:56.779705 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:56.780219 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:56.780220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:56.780221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:56.780297 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:56.780571 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:56.780572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:56.780573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:56.780698 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:56.780788 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:56.781080 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 07:01:56.781167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260927 07:01:56.781453 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:56.770915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.1:45087)
0927 07:01:56.771166 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:56.771213 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:56.771785 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:56.772046 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:56.772050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:56.772362 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:56.772650 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:56.772657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:56.773580 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:56.773581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:56.774865 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:56.774867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:56.775405 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:56.775407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:56.775591 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:56.776253 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:56.776261 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:56.777512 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:56.779795 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:56.779801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:56.779804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:56.780116 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:56.780383 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:56.780383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:56.780384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:56.780493 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:56.780779 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:56.780785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:56.780937 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 07:01:56.781045 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:01:56.781202 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260927 07:01:56.782032 15774 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:56.782152 15774 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:56.782334 15774 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:56.782881 15562 ts_manager.cc:194] Registered new tserver with Master: 75e55c458b24420b83d4f32738af095d (127.3.57.1:42689)
I20260927 07:01:56.783609 15562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:45087
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:01:56.806612 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 07:01:56.931731 15780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:56.931993 15780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:56.932034 15780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:56.933789 15780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:56.933846 15780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:56.933885 15780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:01:56.936090 15780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:56.936517 15780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:56.936823 15780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:56.943607 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:56.943620 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:56.944461 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:56.944971 15780 server_base.cc:1034] running on GCE node
I20260927 07:01:56.945195 15780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:56.945500 15780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:56.947626 15780 hybrid_clock.cc:648] HybridClock initialized: now 1790492516947603 us; error 36 us; skew 500 ppm
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:56.950910 15780 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:01:56.951465 15780 webserver.cc:460] Webserver started at http://127.3.57.2:35405/ using document root <none> and password file <none>
I20260927 07:01:56.951803 15780 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:56.951922 15780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:56.952102 15780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:56 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:56.965932 15780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "d5230f47245a4ba48e66506b616610be"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "f398093f3f2c5e69aba0f04cc288f8b6"
server_key_iv: "72cdef61b0bf2c68dee8b0cc3e503408"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.966337 15780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "d5230f47245a4ba48e66506b616610be"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "f398093f3f2c5e69aba0f04cc288f8b6"
server_key_iv: "72cdef61b0bf2c68dee8b0cc3e503408"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.968662 15780 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:01:56.980870 15797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:56.981101 15780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:56.981187 15780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "d5230f47245a4ba48e66506b616610be"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "f398093f3f2c5e69aba0f04cc288f8b6"
server_key_iv: "72cdef61b0bf2c68dee8b0cc3e503408"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:56.981269 15780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:57.021629 15780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:57.024139 15780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:57.024338 15780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:57.024654 15780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:57.025051 15780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:57.025142 15780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.025219 15780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:57.025281 15780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.031812 15780 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:35161
I20260927 07:01:57.032294 15780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:01:57.033943 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 15780
I20260927 07:01:57.034044 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:01:57.034235 3300 external_mini_cluster.cc:1428] Setting key d9b2231515067443818ada66e8a2d29c
I20260927 07:01:57.037710 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:35161 every 8 connection(s)
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:01:57.050189 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.038649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:35769 (local address 127.3.57.62:44421)
0927 07:01:57.038699 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:57.038702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:57.039539 (+ 837us) server_negotiation.cc:407] Connection header received
0927 07:01:57.040259 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:57.040260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:57.040273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:57.040307 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:57.041153 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.041969 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.043469 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.043838 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.043910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.046612 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:57.046613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:57.046613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:57.046621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:57.048418 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.048917 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.048918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.048918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.049010 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.049321 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.049322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.049323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.049471 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:57.049554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:57.049940 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 07:01:57.050042 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:01:57.050293 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.038355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.2:35769)
0927 07:01:57.039407 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:57.039455 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:57.040130 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:57.040492 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:57.040496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:57.040763 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:57.041032 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.041039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.042085 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.042086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.043356 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.043359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.043991 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.043992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.044137 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.044772 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:57.044783 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:57.046496 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:57.048529 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.048536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.048539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.048827 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.049085 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.049086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.049086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.049233 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.049559 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:57.049565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:57.049745 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 07:01:57.049876 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 07:01:57.050052 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
I20260927 07:01:57.050766 15912 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:57.050856 15912 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.051051 15912 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:57.051657 15562 ts_manager.cc:194] Registered new tserver with Master: d5230f47245a4ba48e66506b616610be (127.3.57.2:35161)
I20260927 07:01:57.052146 15562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:35769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:01:57.071347 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:57.186029 15917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:57.186233 15917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:57.186268 15917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:57.187403 15917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:57.187469 15917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:57.187507 15917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:01:57.188941 15917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:57.189208 15917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:57.189424 15917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:57.195396 15923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.195382 15922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.195482 15925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:57.195698 15917 server_base.cc:1034] running on GCE node
I20260927 07:01:57.195963 15917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:57.196218 15917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:57.197376 15917 hybrid_clock.cc:648] HybridClock initialized: now 1790492517197364 us; error 31 us; skew 500 ppm
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:57.199819 15917 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:01:57.200255 15917 webserver.cc:460] Webserver started at http://127.3.57.3:34567/ using document root <none> and password file <none>
I20260927 07:01:57.200485 15917 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:57.200551 15917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:01:57.200693 15917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:57.221804 15917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59"
format_stamp: "Formatted at 2026-09-27 07:01:57 on dist-test-slave-mk3h"
server_key: "e353808e20581f6aa54f2a16d2888f7f"
server_key_iv: "f33c9cb3c46e7cf8052f51571734e283"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.222352 15917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59"
format_stamp: "Formatted at 2026-09-27 07:01:57 on dist-test-slave-mk3h"
server_key: "e353808e20581f6aa54f2a16d2888f7f"
server_key_iv: "f33c9cb3c46e7cf8052f51571734e283"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.232039 15917 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.001s
I20260927 07:01:57.248803 15932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.249034 15917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:01:57.249121 15917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59"
format_stamp: "Formatted at 2026-09-27 07:01:57 on dist-test-slave-mk3h"
server_key: "e353808e20581f6aa54f2a16d2888f7f"
server_key_iv: "f33c9cb3c46e7cf8052f51571734e283"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.249208 15917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:57.273814 15917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:57.276258 15917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:57.276461 15917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:57.276722 15917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:57.277120 15917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:57.277211 15917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.277283 15917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:57.277343 15917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.286010 15917 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:40915
I20260927 07:01:57.286505 15917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:01:57.287050 16045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:40915 every 8 connection(s)
I20260927 07:01:57.287858 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 15917
I20260927 07:01:57.287957 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:01:57.288151 3300 external_mini_cluster.cc:1428] Setting key c979aaa40a7235408f65003cf8a2a555
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:57.300021 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.287967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:46989 (local address 127.3.57.62:44421)
0927 07:01:57.288013 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:57.288015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:57.289362 (+ 1347us) server_negotiation.cc:407] Connection header received
0927 07:01:57.290084 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:57.290085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:57.290097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:57.290127 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:57.290979 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.291789 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.293290 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.293582 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.293624 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.296239 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:57.296241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:57.296241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:57.296249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:57.297922 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.298377 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.298378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.298379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.298429 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.298727 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.298728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.298728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.298851 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:57.298890 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:57.299848 (+ 958us) server_negotiation.cc:299] Negotiation successful
0927 07:01:57.299927 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:01:57.300729 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.287746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.3:46989)
0927 07:01:57.289239 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:57.289285 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:57.289966 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:57.290309 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:57.290313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:57.290572 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:57.290869 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.290875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.291907 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.291907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.293183 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.293186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.293729 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.293730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.293865 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.294463 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:57.294487 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:57.296127 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:57.298018 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.298025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.298027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.298295 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.298511 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.298512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.298512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.298644 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.298932 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:57.298938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:57.299102 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 07:01:57.300333 (+ 1231us) client_negotiation.cc:240] Negotiation successful
0927 07:01:57.300500 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":1428,"threads_started":1}
I20260927 07:01:57.301318 16046 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:57.301401 16046 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.301584 16046 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:57.301949 15562 ts_manager.cc:194] Registered new tserver with Master: d84f44792fc2451eaaa9d8bcaa8e7f59 (127.3.57.3:40915)
I20260927 07:01:57.302423 15562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:46989
I20260927 07:01:57.310273 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:01:57.311055 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 15643
I20260927 07:01:57.317023 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 15780
I20260927 07:01:57.321750 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 15917
I20260927 07:01:57.327121 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 15530
I20260927 07:01:57.332331 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44421
--webserver_interface=127.3.57.62
--webserver_port=35339
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44421
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:01:57.410561 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:57.410748 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:57.410770 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:57.411948 16049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:01:57.411994 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:57.412009 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:57.412051 16049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:01:57.412070 16049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:01:57.413568 16049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44421
--ranger_config_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=35339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:57.413792 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:57.414018 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:57.418840 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.418840 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.418896 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:57.419104 16049 server_base.cc:1034] running on GCE node
I20260927 07:01:57.419329 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:57.419646 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:57.420807 16049 hybrid_clock.cc:648] HybridClock initialized: now 1790492517420786 us; error 37 us; skew 500 ppm
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:57.423292 16049 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:01:57.423789 16049 webserver.cc:460] Webserver started at http://127.3.57.62:35339/ using document root <none> and password file <none>
I20260927 07:01:57.424005 16049 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:57.424054 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:57.469710 16049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:57.470590 16065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.470798 16049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:57.470907 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "71a65ffebfcd4d63a5f5af97301d2394"
format_stamp: "Formatted at 2026-09-27 07:01:55 on dist-test-slave-mk3h"
server_key: "90194fddcc7065b541a25f6a7c6e956c"
server_key_iv: "57eb28c0621cab6487676aaa181d1bfb"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.471280 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:57.485270 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:57.486729 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:57.486883 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:57.490953 16049 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:44421
I20260927 07:01:57.491001 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:44421 every 8 connection(s)
I20260927 07:01:57.491338 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:01:57.492487 16118 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:01:57.493116 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Bootstrap starting.
I20260927 07:01:57.497236 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16049
I20260927 07:01:57.497550 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:42689
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:44421
--webserver_port=35761
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:01:57.501343 16118 log.cc:824] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:57.502936 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:01:57.503214 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Bootstrap complete.
I20260927 07:01:57.505616 16118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:57.505832 16118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a65ffebfcd4d63a5f5af97301d2394, State: Initialized, Role: FOLLOWER
I20260927 07:01:57.505982 16118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:57.506057 16118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:01:57.506125 16118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:01:57.506172 16118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:01:57.507105 16118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:57.507186 16118 leader_election.cc:302] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a65ffebfcd4d63a5f5af97301d2394; no voters:
I20260927 07:01:57.507375 16118 leader_election.cc:288] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:01:57.507406 16122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:01:57.507520 16122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [term 2 LEADER]: Becoming Leader. State: Replica: 71a65ffebfcd4d63a5f5af97301d2394, State: Running, Role: LEADER
I20260927 07:01:57.507642 16118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:01:57.507695 16122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } }
I20260927 07:01:57.508036 16123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "71a65ffebfcd4d63a5f5af97301d2394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } } }
I20260927 07:01:57.508121 16123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:57.508327 16124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a65ffebfcd4d63a5f5af97301d2394. Latest consensus state: current_term: 2 leader_uuid: "71a65ffebfcd4d63a5f5af97301d2394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a65ffebfcd4d63a5f5af97301d2394" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44421 } } }
I20260927 07:01:57.508406 16124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394 [sys.catalog]: This master's current role is: LEADER
I20260927 07:01:57.508508 16128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:01:57.508991 16128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:01:57.509173 16128 catalog_manager.cc:1259] Loaded cluster ID: 0923a7b316514d209af27905e6b86c53
I20260927 07:01:57.509254 16128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:01:57.509320 16118 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:01:57.512905 16128 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:01:57.513384 16128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Loaded TSK: 0
I20260927 07:01:57.513690 16128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:01:57.613894 16120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:57.614176 16120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:57.614223 16120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:57.615898 16120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:57.615962 16120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:57.616019 16120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:01:57.618247 16120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:42689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=35761
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:57.618476 16120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:57.618736 16120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:57.624979 16158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.624948 16155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:57.627600 16120 server_base.cc:1034] running on GCE node
W20260927 07:01:57.627710 16156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:57.627951 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:57.628245 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:57.629467 16120 hybrid_clock.cc:648] HybridClock initialized: now 1790492517629442 us; error 38 us; skew 500 ppm
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:57.632634 16120 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:01:57.633193 16120 webserver.cc:460] Webserver started at http://127.3.57.1:35761/ using document root <none> and password file <none>
I20260927 07:01:57.633500 16120 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:57.633586 16120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:57.652537 16120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:01:57.653611 16166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.653765 16120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:01:57.653837 16120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "75e55c458b24420b83d4f32738af095d"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "0b3b7d787fdb5107734326af4af8ca38"
server_key_iv: "c4e9ddf06d1d9355843fb584da5be044"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.654270 16120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:57.687193 16120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:57.689672 16120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:57.689859 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:57.690119 16120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:57.690480 16120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:57.690529 16120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.690603 16120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:57.690634 16120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.698448 16120 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:42689
I20260927 07:01:57.698825 16279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:42689 every 8 connection(s)
I20260927 07:01:57.698940 16120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:01:57.703773 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16120
I20260927 07:01:57.704083 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:35161
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:44421
--webserver_port=35405
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:57.717522 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.702720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:41241 (local address 127.3.57.62:44421)
0927 07:01:57.702936 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:57.702946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:57.703208 (+ 262us) server_negotiation.cc:407] Connection header received
0927 07:01:57.704027 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:57.704034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:57.704092 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:57.704210 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:57.705073 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.706511 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.709246 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.709982 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.710073 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.712959 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:57.712965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:57.712965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:57.712979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:57.715130 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.715707 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.715711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.715711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.715825 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.716144 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.716145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.716145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.716499 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:57.716598 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:57.717164 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 07:01:57.717297 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260927 07:01:57.718060 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.702708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.1:41241)
0927 07:01:57.703034 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:57.703075 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:57.703736 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:57.704406 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:57.704409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:57.704644 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:57.704935 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.704941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.707680 (+ 2739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.707681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.709137 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.709141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.710278 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.710279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.710426 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.711016 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:57.711027 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:57.712824 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:57.715232 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.715239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.715241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.715591 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.715907 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.715907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.715908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.716041 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.716629 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:57.716634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:57.716776 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 07:01:57.717695 (+ 919us) client_negotiation.cc:240] Negotiation successful
0927 07:01:57.717845 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":141,"threads_started":1}
I20260927 07:01:57.721809 16280 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:57.721935 16280 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.722116 16280 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:57.722956 16280 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:44421 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:01:57.723026 16280 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.723086 16280 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:57.723476 16280 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.723642 16280 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:57.723958 16280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:01:57.831030 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:57.831310 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:57.831346 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:57.833040 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:57.833101 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:57.833150 16284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:01:57.835307 16284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:35161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=35405
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:57.835587 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:57.835855 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:57.841559 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.841545 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:57.841701 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:57.841883 16284 server_base.cc:1034] running on GCE node
I20260927 07:01:57.842085 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:57.842345 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:57.843524 16284 hybrid_clock.cc:648] HybridClock initialized: now 1790492517843506 us; error 33 us; skew 500 ppm
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:57.846330 16284 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:01:57.846849 16284 webserver.cc:460] Webserver started at http://127.3.57.2:35405/ using document root <none> and password file <none>
I20260927 07:01:57.847110 16284 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:57.847195 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:57.864804 16284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:01:57.867970 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.868150 16284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:01:57.868243 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "d5230f47245a4ba48e66506b616610be"
format_stamp: "Formatted at 2026-09-27 07:01:56 on dist-test-slave-mk3h"
server_key: "f398093f3f2c5e69aba0f04cc288f8b6"
server_key_iv: "72cdef61b0bf2c68dee8b0cc3e503408"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:57.868626 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:57.881711 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:57.884792 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:57.884943 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:57.885187 16284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:57.885537 16284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:57.885584 16284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.885612 16284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:57.885677 16284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:57.893131 16284 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:35161
I20260927 07:01:57.893193 16413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:35161 every 8 connection(s)
I20260927 07:01:57.893570 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:01:57 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:57.900723 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16284
I20260927 07:01:57.901026 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.57.20:48247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:40915
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:44421
--webserver_port=34567
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:01:57.903082 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.894480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:52049 (local address 127.3.57.62:44421)
0927 07:01:57.894545 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:57.894547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:57.894760 (+ 213us) server_negotiation.cc:407] Connection header received
0927 07:01:57.895276 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:57.895278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:57.895291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:57.895340 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:57.895975 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.896484 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.897497 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.897698 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.897783 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.899799 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:57.899801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:57.899801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:57.899809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:57.901417 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.901923 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.901924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.901925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.901990 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:57.902290 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:57.902291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:57.902291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:57.902436 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:57.902507 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:57.902871 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 07:01:57.902963 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:01:57.903226 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:57.894447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.2:52049)
0927 07:01:57.894637 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:57.894672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:57.895166 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:57.895405 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:57.895409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:57.895691 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:57.895877 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.895882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.896578 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.896579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.897407 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:57.897409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:57.897809 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:57.897810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:57.897920 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:57.898427 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:57.898434 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:57.899708 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:57.901521 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.901527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.901529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.901835 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.902071 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:57.902071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:57.902072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:57.902208 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:57.902538 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:57.902545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:57.902707 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 07:01:57.902830 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 07:01:57.902986 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260927 07:01:57.903949 16414 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:57.904067 16414 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.904250 16414 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:57.904919 16414 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:44421 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:01:57.904989 16414 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.905068 16414 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:57.905436 16414 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:57.905514 16414 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:57.905851 16414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:01:57.997159 16417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:01:57.997364 16417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:01:57.997402 16417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:01:57.998653 16417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:01:57.998723 16417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:01:57.998767 16417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:01:58.001207 16417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.57.20:48247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:40915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=34567
--tserver_master_addrs=127.3.57.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:01:58.001490 16417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:01:58.001749 16417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:01:58.006825 16425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:58.007129 16422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:01:58.007714 16423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:01:58.008220 16417 server_base.cc:1034] running on GCE node
I20260927 07:01:58.008437 16417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:01:58.008729 16417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:01:58.009891 16417 hybrid_clock.cc:648] HybridClock initialized: now 1790492518009867 us; error 40 us; skew 500 ppm
Sep 27 07:01:58 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:58.012944 16417 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:01:58.013451 16417 webserver.cc:460] Webserver started at http://127.3.57.3:34567/ using document root <none> and password file <none>
I20260927 07:01:58.013716 16417 fs_manager.cc:359] Metadata directory not provided
I20260927 07:01:58.013782 16417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:01:58 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:58.032436 16417 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:01:58.033329 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:58.033494 16417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:01:58.033602 16417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59"
format_stamp: "Formatted at 2026-09-27 07:01:57 on dist-test-slave-mk3h"
server_key: "e353808e20581f6aa54f2a16d2888f7f"
server_key_iv: "f33c9cb3c46e7cf8052f51571734e283"
server_key_version: "kuduclusterkey@0"
I20260927 07:01:58.033986 16417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:01:58.046190 16417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:01:58.048403 16417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:01:58.048602 16417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:01:58.048851 16417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:01:58.049139 16417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:01:58.049193 16417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:58.049233 16417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:01:58.049261 16417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:01:58.055527 16417 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:40915
I20260927 07:01:58.055660 16547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:40915 every 8 connection(s)
I20260927 07:01:58.055943 16417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:01:58.057096 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16417
Sep 27 07:01:58 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:01:58.065680 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:58.057505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.3.57.62:44421)
0927 07:01:58.058569 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:58.058571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:58.058577 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:01:58.058596 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:58.058597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:58.058606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:58.058664 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:58.059420 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.059972 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.060892 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.061092 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.061158 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:58.062472 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:58.062474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:58.062474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:58.062481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:58.064487 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:58.064860 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:58.064861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:58.064861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:58.064921 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:58.065134 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:58.065135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:58.065135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:58.065255 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:58.065430 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:58.065494 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 07:01:58.065583 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":69,"threads_started":1}
I20260927 07:01:58.066632 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:58.056607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44421 (local address 127.3.57.3:48399)
0927 07:01:58.056760 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:58.056805 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:58.057457 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:58.057804 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:58.057808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:58.058019 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:01:58.058286 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:58.058291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.059099 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.059100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:58.060369 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:58.060372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.060881 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.060883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:58.061014 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:58.061563 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:01:58.061572 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:01:58.063039 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:01:58.065037 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:58.065043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:58.065045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:58.065347 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:58.065549 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:01:58.065550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:01:58.065550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:01:58.065668 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:01:58.065973 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:01:58.065979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:01:58.066134 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 07:01:58.066239 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:01:58.066422 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260927 07:01:58.066936 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:58.056619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:48399 (local address 127.3.57.62:44421)
0927 07:01:58.056678 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:58.056681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:58.056901 (+ 220us) server_negotiation.cc:407] Connection header received
0927 07:01:58.057595 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:58.057596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:58.057608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:58.057637 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:58.058369 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.059004 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.060455 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:58.060774 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:58.060844 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:58.063126 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:58.063127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:58.063128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:58.063135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:58.064962 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:58.065419 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:58.065420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:58.065420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:58.065471 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:58.065739 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:58.065740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:58.065740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:58.065886 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:58.065946 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:58.066804 (+ 858us) server_negotiation.cc:299] Negotiation successful
0927 07:01:58.066881 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:01:58.067255 16548 heartbeater.cc:347] Connected to a master server at 127.3.57.62:44421
I20260927 07:01:58.067348 16548 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:58.067521 16548 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:58.068181 16548 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:44421 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:01:58.068261 16548 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:58.068328 16548 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:58.068643 16548 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:58.068713 16548 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
W20260927 07:01:58.069098 16548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 07:01:58 dist-test-slave-mk3h krb5kdc[15086](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:01:58.724159 16280 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:58.724308 16280 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:58.724663 16082 ts_manager.cc:194] Registered new tserver with Master: 75e55c458b24420b83d4f32738af095d (127.3.57.1:42689)
I20260927 07:01:58.725636 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:41241
I20260927 07:01:58.906035 16414 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:58.906176 16414 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:58.906436 16082 ts_manager.cc:194] Registered new tserver with Master: d5230f47245a4ba48e66506b616610be (127.3.57.2:35161)
I20260927 07:01:58.906962 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:52049
Sep 27 07:01:58 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
I20260927 07:01:59.069283 16548 heartbeater.cc:464] Registering TS with master...
I20260927 07:01:59.069430 16548 heartbeater.cc:511] Master 127.3.57.62:44421 requested a full tablet report, sending...
I20260927 07:01:59.069700 16082 ts_manager.cc:194] Registered new tserver with Master: d84f44792fc2451eaaa9d8bcaa8e7f59 (127.3.57.3:40915)
I20260927 07:01:59.070223 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:48399
I20260927 07:01:59.078799 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:01:59.091495 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.079800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.3.57.62:44421)
0927 07:01:59.079930 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.079932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.083751 (+ 3819us) server_negotiation.cc:407] Connection header received
0927 07:01:59.083783 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.083784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.083803 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.083888 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:01:59.084564 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.085360 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.086426 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.086740 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.086820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.088390 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:01:59.088391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:01:59.088391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:01:59.088399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:01:59.090225 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:59.090636 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:59.090638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:59.090638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:59.090712 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:01:59.090930 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:01:59.090931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:01:59.090931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:01:59.091062 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:01:59.091162 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.091308 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.091391 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260927 07:01:59.093864 16082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:01:59.096061 16082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:01:59.103824 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.098993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40915 (local address 127.0.0.1:45420)
0927 07:01:59.099651 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.099659 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:59.099703 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:59.099955 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:59.099958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:59.099961 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:59.100023 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.100024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.101213 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.101214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.102439 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.102443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.103484 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.103485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.103560 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.103573 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:01:59.103676 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:01:59.103738 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":59,"threads_started":1}
I20260927 07:01:59.104453 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.099167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.3.57.3:40915)
0927 07:01:59.099348 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.099352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.099757 (+ 405us) server_negotiation.cc:407] Connection header received
0927 07:01:59.099828 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.099830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.099846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.099916 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:59.100107 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.101115 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.102544 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.103314 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.103399 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.103450 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.104330 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.104376 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":50,"threads_started":1}
I20260927 07:01:59.106001 16482 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f (DEFAULT_TABLE table=test-table [id=aba6a291328f4549b567759557437397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:59.106324 16482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5795443d0843b8a2ba2b23ad147c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:59.109211 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.098800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:42689 (local address 127.0.0.1:57748)
0927 07:01:59.099111 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.099153 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:59.099218 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:59.105505 (+ 6287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:59.105505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:59.105507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:59.105545 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.105547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.106786 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.106787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.107954 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.107957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.109010 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.109011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.109071 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.109075 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:01:59.109103 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 07:01:59.109137 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":74,"threads_started":1}
I20260927 07:01:59.109622 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.098904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.3.57.1:42689)
0927 07:01:59.105313 (+ 6409us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.105317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.105328 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:01:59.105362 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.105364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.105395 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.105611 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:59.105692 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.106683 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.108064 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.108915 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.109414 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.109482 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.109507 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.109545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6375,"thread_start_us":63,"threads_started":1}
I20260927 07:01:59.110236 16202 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f (DEFAULT_TABLE table=test-table [id=aba6a291328f4549b567759557437397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:59.110545 16202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5795443d0843b8a2ba2b23ad147c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:59.112154 16581 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Bootstrap starting.
I20260927 07:01:59.112766 16581 tablet_bootstrap.cc:654] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:59.113085 16581 log.cc:824] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:59.114892 16581 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: No bootstrap required, opened a new log
I20260927 07:01:59.115010 16581 ts_tablet_manager.cc:1397] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:01:59.117048 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.099350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:35161 (local address 127.0.0.1:35758)
0927 07:01:59.104219 (+ 4869us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.104228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:59.104259 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:59.113600 (+ 9341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:59.113601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:59.113603 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:59.113633 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.113634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.114892 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.114893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.115939 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.115942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.116862 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.116863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.116928 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.116931 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:01:59.116957 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 07:01:59.116991 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4858,"thread_start_us":79,"threads_started":1}
I20260927 07:01:59.117569 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.099442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.3.57.2:35161)
0927 07:01:59.113421 (+ 13979us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.113425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.113437 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:01:59.113470 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.113472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.113487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.113708 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:59.113785 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.114792 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.116043 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.116764 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.117360 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.117423 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.117446 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.117483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13941,"thread_start_us":63,"threads_started":1}
I20260927 07:01:59.117589 16581 raft_consensus.cc:348] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.117834 16581 raft_consensus.cc:374] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:59.117902 16581 raft_consensus.cc:729] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84f44792fc2451eaaa9d8bcaa8e7f59, State: Initialized, Role: FOLLOWER
I20260927 07:01:59.118067 16581 consensus_queue.cc:260] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.118263 16332 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f (DEFAULT_TABLE table=test-table [id=aba6a291328f4549b567759557437397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:01:59.118436 16581 ts_tablet_manager.cc:1428] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:01:59.118544 16548 heartbeater.cc:503] Master 127.3.57.62:44421 was elected leader, sending a full tablet report...
I20260927 07:01:59.118554 16332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5795443d0843b8a2ba2b23ad147c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:01:59.119699 16583 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Bootstrap starting.
I20260927 07:01:59.120378 16583 tablet_bootstrap.cc:654] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:59.120771 16583 log.cc:824] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:59.122045 16586 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Bootstrap starting.
I20260927 07:01:59.122501 16583 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: No bootstrap required, opened a new log
I20260927 07:01:59.122619 16583 ts_tablet_manager.cc:1397] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:01:59.122666 16586 tablet_bootstrap.cc:654] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Neither blocks nor log segments found. Creating new log.
I20260927 07:01:59.123184 16586 log.cc:824] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Log is configured to *not* fsync() on all Append() calls
I20260927 07:01:59.123884 16586 tablet_bootstrap.cc:492] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: No bootstrap required, opened a new log
I20260927 07:01:59.123942 16586 ts_tablet_manager.cc:1397] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:01:59.125183 16583 raft_consensus.cc:348] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.125334 16583 raft_consensus.cc:374] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:59.125363 16583 raft_consensus.cc:729] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e55c458b24420b83d4f32738af095d, State: Initialized, Role: FOLLOWER
I20260927 07:01:59.125490 16583 consensus_queue.cc:260] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.125746 16583 ts_tablet_manager.cc:1428] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:01:59.125857 16280 heartbeater.cc:503] Master 127.3.57.62:44421 was elected leader, sending a full tablet report...
I20260927 07:01:59.126223 16586 raft_consensus.cc:348] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.126353 16586 raft_consensus.cc:374] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:01:59.126377 16586 raft_consensus.cc:729] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5230f47245a4ba48e66506b616610be, State: Initialized, Role: FOLLOWER
I20260927 07:01:59.126490 16586 consensus_queue.cc:260] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.126698 16586 ts_tablet_manager.cc:1428] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:01:59.126842 16414 heartbeater.cc:503] Master 127.3.57.62:44421 was elected leader, sending a full tablet report...
W20260927 07:01:59.144855 16415 tablet.cc:2369] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:01:59 dist-test-slave-mk3h krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for HTTP/127.3.57.20@KRBTEST.COM
W20260927 07:01:59.200289 16281 tablet.cc:2369] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:01:59.283386 16588 raft_consensus.cc:482] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:01:59.283576 16588 raft_consensus.cc:504] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.284015 16588 leader_election.cc:288] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5230f47245a4ba48e66506b616610be (127.3.57.2:35161), d84f44792fc2451eaaa9d8bcaa8e7f59 (127.3.57.3:40915)
I20260927 07:01:59.299345 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.284395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:60421 (local address 127.3.57.2:35161)
0927 07:01:59.284498 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.284500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.288191 (+ 3691us) server_negotiation.cc:407] Connection header received
0927 07:01:59.295703 (+ 7512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.295705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.295718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.295747 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:59.295967 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.296782 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.298331 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.298934 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.299011 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.299022 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.299228 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.299272 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:01:59.299856 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.284177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:35161 (local address 127.3.57.1:60421)
0927 07:01:59.288100 (+ 3923us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.288113 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:59.295594 (+ 7481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:59.295815 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:59.295816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:59.295817 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:59.295897 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.295899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.296898 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.296899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.298209 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.298213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.299076 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.299077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.299143 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.299147 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:01:59.299707 (+ 560us) client_negotiation.cc:240] Negotiation successful
0927 07:01:59.299755 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3901,"thread_start_us":123,"threads_started":1}
I20260927 07:01:59.300307 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5795443d0843b8a2ba2b23ad147c0f" candidate_uuid: "75e55c458b24420b83d4f32738af095d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5230f47245a4ba48e66506b616610be" is_pre_election: true
I20260927 07:01:59.300462 16351 raft_consensus.cc:2454] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e55c458b24420b83d4f32738af095d in term 0.
I20260927 07:01:59.300688 16169 leader_election.cc:302] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e55c458b24420b83d4f32738af095d, d5230f47245a4ba48e66506b616610be; no voters:
I20260927 07:01:59.300984 16588 raft_consensus.cc:2781] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:01:59.301122 16588 raft_consensus.cc:482] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:01:59.301147 16588 raft_consensus.cc:3037] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:59.301556 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.284442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:58117 (local address 127.3.57.3:40915)
0927 07:01:59.284505 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.284507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.298408 (+ 13901us) server_negotiation.cc:407] Connection header received
0927 07:01:59.298537 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.298538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.298551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.298580 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:01:59.298788 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.299377 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.300620 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.301178 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.301224 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.301233 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.301440 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.301479 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 07:01:59.301780 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.284209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40915 (local address 127.3.57.1:58117)
0927 07:01:59.298352 (+ 14143us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.298363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:01:59.298452 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:01:59.298640 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:01:59.298640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:01:59.298642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:01:59.298704 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.298705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.299473 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.299474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.300515 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:01:59.300518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.301291 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.301292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:01:59.301358 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.301361 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:01:59.301665 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 07:01:59.301705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14126,"thread_start_us":91,"threads_started":1}
I20260927 07:01:59.302060 16588 raft_consensus.cc:504] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.302047 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5795443d0843b8a2ba2b23ad147c0f" candidate_uuid: "75e55c458b24420b83d4f32738af095d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" is_pre_election: true
I20260927 07:01:59.302208 16588 leader_election.cc:288] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [CANDIDATE]: Term 1 election: Requested vote from peers d5230f47245a4ba48e66506b616610be (127.3.57.2:35161), d84f44792fc2451eaaa9d8bcaa8e7f59 (127.3.57.3:40915)
I20260927 07:01:59.302206 16502 raft_consensus.cc:2454] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e55c458b24420b83d4f32738af095d in term 0.
I20260927 07:01:59.302561 16502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5795443d0843b8a2ba2b23ad147c0f" candidate_uuid: "75e55c458b24420b83d4f32738af095d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59"
I20260927 07:01:59.302632 16502 raft_consensus.cc:3037] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:59.302699 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5795443d0843b8a2ba2b23ad147c0f" candidate_uuid: "75e55c458b24420b83d4f32738af095d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5230f47245a4ba48e66506b616610be"
I20260927 07:01:59.302772 16351 raft_consensus.cc:3037] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:01:59.303488 16502 raft_consensus.cc:2454] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e55c458b24420b83d4f32738af095d in term 1.
I20260927 07:01:59.303614 16351 raft_consensus.cc:2454] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e55c458b24420b83d4f32738af095d in term 1.
I20260927 07:01:59.303776 16170 leader_election.cc:302] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e55c458b24420b83d4f32738af095d, d84f44792fc2451eaaa9d8bcaa8e7f59; no voters:
I20260927 07:01:59.303865 16588 raft_consensus.cc:2781] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:01:59.303997 16588 raft_consensus.cc:686] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [term 1 LEADER]: Becoming Leader. State: Replica: 75e55c458b24420b83d4f32738af095d, State: Running, Role: LEADER
I20260927 07:01:59.304083 16588 consensus_queue.cc:237] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } }
I20260927 07:01:59.304906 16081 catalog_manager.cc:5495] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e55c458b24420b83d4f32738af095d (127.3.57.1). New cstate: current_term: 1 leader_uuid: "75e55c458b24420b83d4f32738af095d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e55c458b24420b83d4f32738af095d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 42689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 } health_report { overall_health: UNKNOWN } } }
W20260927 07:01:59.307360 16549 tablet.cc:2369] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:01:59.340780 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.337542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.3.57.1:42689)
0927 07:01:59.337585 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:01:59.337586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:01:59.337594 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:01:59.337635 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:01:59.337636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:01:59.337652 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:01:59.337746 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:01:59.337917 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.338624 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.339726 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:01:59.340033 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:01:59.340096 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:01:59.340286 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:01:59.340405 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:01:59.340479 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:01:59.340691 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 07:01:59.340709 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:01:59.342916 16502 raft_consensus.cc:1261] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 1 FOLLOWER]: Refusing update from remote peer 75e55c458b24420b83d4f32738af095d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:01:59.343319 16594 consensus_queue.cc:1035] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84f44792fc2451eaaa9d8bcaa8e7f59" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:01:59.345444 16351 raft_consensus.cc:1261] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 1 FOLLOWER]: Refusing update from remote peer 75e55c458b24420b83d4f32738af095d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:01:59.345775 16594 consensus_queue.cc:1035] T 8e5795443d0843b8a2ba2b23ad147c0f P 75e55c458b24420b83d4f32738af095d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5230f47245a4ba48e66506b616610be" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:01:59.349210 16597 mvcc.cc:205] Tried to move back new op lower bound from 7333857359226179584 to 7333857359070060544. Current Snapshot: MvccSnapshot[applied={T|T < 7333857359226179584}]
I20260927 07:01:59.350209 16600 mvcc.cc:205] Tried to move back new op lower bound from 7333857359226179584 to 7333857359070060544. Current Snapshot: MvccSnapshot[applied={T|T < 7333857359226179584}]
I20260927 07:01:59.351954 16598 mvcc.cc:205] Tried to move back new op lower bound from 7333857359226179584 to 7333857359070060544. Current Snapshot: MvccSnapshot[applied={T|T < 7333857359226179584}]
I20260927 07:01:59.354465 16081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:01:59.354557 16081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:01:59.355338 16081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71a65ffebfcd4d63a5f5af97301d2394: Sending DeleteTablet for 3 replicas of tablet 8e5795443d0843b8a2ba2b23ad147c0f
I20260927 07:01:59.355743 16202 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:59 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57748
I20260927 07:01:59.355942 16482 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:59 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:45420
I20260927 07:01:59.356325 16332 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5795443d0843b8a2ba2b23ad147c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:01:59 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:35758
I20260927 07:01:59.356938 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16120
I20260927 07:01:59.363895 16608 tablet_replica.cc:330] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be: stopping tablet replica
I20260927 07:01:59.364068 16608 raft_consensus.cc:2229] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:01:59.364274 16608 raft_consensus.cc:2258] T 8e5795443d0843b8a2ba2b23ad147c0f P d5230f47245a4ba48e66506b616610be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:01:59.364408 16607 tablet_replica.cc:330] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: stopping tablet replica
I20260927 07:01:59.364612 16607 raft_consensus.cc:2229] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:01:59.364773 16607 raft_consensus.cc:2258] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:01:59.365272 16607 ts_tablet_manager.cc:1905] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:01:59.367236 16607 ts_tablet_manager.cc:1918] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:01:59.367349 16607 log.cc:1196] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/8e5795443d0843b8a2ba2b23ad147c0f
I20260927 07:01:59.367841 16607 ts_tablet_manager.cc:1939] T 8e5795443d0843b8a2ba2b23ad147c0f P d84f44792fc2451eaaa9d8bcaa8e7f59: Deleting consensus metadata
I20260927 07:01:59.368330 16069 catalog_manager.cc:4841] TS d84f44792fc2451eaaa9d8bcaa8e7f59 (127.3.57.3:40915): tablet 8e5795443d0843b8a2ba2b23ad147c0f (table test-table [id=aba6a291328f4549b567759557437397]) successfully deleted
W20260927 07:01:59.368714 16069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:42689 (error 108)
I20260927 07:01:59.369536 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:01:59.369409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:42689 (local address 127.0.0.1:57772)
0927 07:01:59.369453 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:01:59.369490 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 07:01:59.369527 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16284
W20260927 07:01:59.369810 16069 catalog_manager.cc:4584] TS 75e55c458b24420b83d4f32738af095d (127.3.57.1:42689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e5795443d0843b8a2ba2b23ad147c0f: Network error: Client connection negotiation failed: client connection to 127.3.57.1:42689: connect: Connection refused (error 111)
I20260927 07:01:59.376047 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16417
I20260927 07:01:59.383869 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16049
I20260927 07:01:59.390355 3300 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:01:59.734534 3300 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:01:59.734630 3300 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:02:00.079236 3300 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:02:00.079 UTC [15116] LOG: received smart shutdown request
2026-09-27 07:02:00.083 UTC [15116] LOG: background worker "logical replication launcher" (PID 15125) exited with exit code 1
2026-09-27 07:02:00.083 UTC [15120] LOG: shutting down
2026-09-27 07:02:00.242 UTC [15116] LOG: database system is shut down
2026-09-27T07:02:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46467 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:02:00 dist-test-slave-mk3h krb5kdc[16618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:02:00 dist-test-slave-mk3h krb5kdc[16618](info): set up 2 sockets
Sep 27 07:02:00 dist-test-slave-mk3h krb5kdc[16618](info): commencing operation
krb5kdc: starting...
W20260927 07:02:01.399422 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:02:01.508301 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40713
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:01.585429 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:01.585637 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:01.585660 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:01.586719 16631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:01.586759 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:01.586773 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:01.586786 16631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:01.586798 16631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:01.588256 16631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:01.588435 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:01.588601 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:01.593120 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:01.593115 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:01.593115 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:01.593361 16631 server_base.cc:1034] running on GCE node
I20260927 07:02:01.593580 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:01.593904 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:01.595067 16631 hybrid_clock.cc:648] HybridClock initialized: now 1790492521595054 us; error 42 us; skew 500 ppm
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:01.601229 16631 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:01.601693 16631 webserver.cc:460] Webserver started at http://127.3.57.62:34953/ using document root <none> and password file <none>
I20260927 07:02:01.601926 16631 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:01.601994 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:01.602159 16631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:01.603190 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "cf9fc3e156a14478bdeafa6337fd64f5"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "bc5d8193120a898b8053e4a443bd0bc5"
server_key_iv: "875e758e4147cde84f4e46c30ea6831e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.603585 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "cf9fc3e156a14478bdeafa6337fd64f5"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "bc5d8193120a898b8053e4a443bd0bc5"
server_key_iv: "875e758e4147cde84f4e46c30ea6831e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.604849 16631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:01.605543 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:01.605710 16631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:01.605799 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "cf9fc3e156a14478bdeafa6337fd64f5"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "bc5d8193120a898b8053e4a443bd0bc5"
server_key_iv: "875e758e4147cde84f4e46c30ea6831e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.605885 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:01.618240 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:01.620482 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:01.620635 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:01.624758 16631 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:40713
I20260927 07:02:01.624825 16698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:40713 every 8 connection(s)
I20260927 07:02:01.625155 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:01.625950 16699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:01.627449 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.632525 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16631
I20260927 07:02:01.632643 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:02:01.632831 3300 external_mini_cluster.cc:1428] Setting key 9677abb93820a3a1aa79ce8e699721ef
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:01.637110 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.628095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40713 (local address 127.0.0.1:47640)
0927 07:02:01.628251 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.628289 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.628842 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.629096 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.629098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.629320 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.629455 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.629459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.630430 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.630431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.631074 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.631076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.631537 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.631538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.631654 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.632044 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.632052 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.633687 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.635759 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.635764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.635766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.635949 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.636197 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.636198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.636198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.636271 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.636692 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.636698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.636771 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.636842 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.636968 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260927 07:02:01.637228 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.628130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.3.57.62:40713)
0927 07:02:01.628298 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.628302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.628379 (+ 77us) server_negotiation.cc:407] Connection header received
0927 07:02:01.628968 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.628970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.628986 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.629071 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.629538 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.630350 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.631162 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.631463 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.631587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.633764 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.633770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.633771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.633776 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.635671 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.636027 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.636030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.636031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.636130 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.636332 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.636333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.636333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.636613 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.636714 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.636903 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.637049 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260927 07:02:01.638118 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.638463 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.638356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57398)
0927 07:02:01.638400 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.638440 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:02:01.638613 16649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
I20260927 07:02:01.639019 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.638928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57412)
0927 07:02:01.638975 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.638998 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:02:01.639158 16699 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:46637: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.57.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
I20260927 07:02:01.656545 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:46637
--webserver_interface=127.3.57.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:01.688370 16699 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } attempt: 1
I20260927 07:02:01.688822 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.688684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57422)
0927 07:02:01.688749 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.688786 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 07:02:01.689154 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.689058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57438)
0927 07:02:01.689109 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.689138 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 07:02:01.689236 16699 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:46637: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
W20260927 07:02:01.735039 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:01.735340 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:01.735375 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:01.737350 16706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:01.737419 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:01.737445 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:01.737494 16706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:01.737522 16706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:01.739785 16706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:46637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.57.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:01.740024 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:01.740296 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:01.744964 16711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:01.745004 16712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:01.745175 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:01.745134 16706 server_base.cc:1034] running on GCE node
I20260927 07:02:01.745463 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:01.745766 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:01.746933 16706 hybrid_clock.cc:648] HybridClock initialized: now 1790492521746925 us; error 38 us; skew 500 ppm
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:01.749293 16706 init.cc:377] Logged in from keytab as kudu/127.3.57.61@KRBTEST.COM (short username kudu)
I20260927 07:02:01.749749 16706 webserver.cc:460] Webserver started at http://127.3.57.61:36321/ using document root <none> and password file <none>
I20260927 07:02:01.749984 16706 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:01.750051 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:01.750216 16706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:01.751207 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data/instance:
uuid: "e818826c0457497695417c1a44553d6e"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "19380897a8e4b68725e3f52e4cf631fe"
server_key_iv: "e3b0916d280198ac42586989734b4ff8"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.751595 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal/instance:
uuid: "e818826c0457497695417c1a44553d6e"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "19380897a8e4b68725e3f52e4cf631fe"
server_key_iv: "e3b0916d280198ac42586989734b4ff8"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.752856 16706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:02:01.753522 16721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:01.753705 16706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:02:01.753803 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "e818826c0457497695417c1a44553d6e"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "19380897a8e4b68725e3f52e4cf631fe"
server_key_iv: "e3b0916d280198ac42586989734b4ff8"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.753892 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:01.757362 16699 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } attempt: 2
I20260927 07:02:01.757710 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.757589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57442)
0927 07:02:01.757643 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.757673 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 07:02:01.757997 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.757910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57452)
0927 07:02:01.757951 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.757975 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 07:02:01.758083 16699 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:46637: Network error: Client connection negotiation failed: client connection to 127.3.57.61:46637: connect: Connection refused (error 111)
I20260927 07:02:01.768672 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:01.770390 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:01.770547 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:01.774771 16706 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.61:46637
I20260927 07:02:01.774799 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.61:46637 every 8 connection(s)
I20260927 07:02:01.775120 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
I20260927 07:02:01.775978 16774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:01.777355 16774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.780596 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16706
I20260927 07:02:01.780681 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-1/wal/instance
I20260927 07:02:01.780857 3300 external_mini_cluster.cc:1428] Setting key 331222bd82ce9cad0fc9df0466dc1bd4
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:01.786542 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.777974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.3.57.62:40713)
0927 07:02:01.778048 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.778050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.778291 (+ 241us) server_negotiation.cc:407] Connection header received
0927 07:02:01.778803 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.778804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.778812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.778863 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.779445 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.779970 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.781139 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.781479 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.781556 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.783801 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.783802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.783803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.783810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.785233 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.785630 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.785631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.785632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.785689 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.785930 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.785931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.785931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.786037 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.786107 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.786378 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.786457 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 07:02:01.786725 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.777975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40713 (local address 127.0.0.1:47648)
0927 07:02:01.778151 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.778195 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.778718 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.778923 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.778926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.779175 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.779369 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.779373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.780062 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.780063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.781056 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.781058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.781578 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.781579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.781682 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.782189 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.782199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.783710 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.785315 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.785321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.785323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.785541 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.785753 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.785754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.785754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.785865 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.786111 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.786118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.786247 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.786341 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.786488 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260927 07:02:01.787225 16774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.57.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.60@KRBTEST.COM" created.
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:02:01.794493 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.787534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.3.57.61:46637)
0927 07:02:01.787676 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.787680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.787701 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:02:01.787758 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.787760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.787772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.787829 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.788084 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.788809 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.789516 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.789824 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.789914 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.791674 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.791681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.791681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.791689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.793206 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.793596 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.793598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.793599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.793647 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.793878 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.793880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.793880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.794028 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.794186 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.794339 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.794417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260927 07:02:01.794900 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.787457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57458)
0927 07:02:01.787502 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.787659 (+ 157us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.787702 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.787840 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.787841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.787975 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.788013 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.788014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.788896 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.788897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.789430 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.789432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.789913 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.789914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.789984 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.790324 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.790328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.791599 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.793288 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.793289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.793289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.793516 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.793719 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.793719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.793720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.793790 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.794165 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.794166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.794220 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.794311 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.794502 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":88}
I20260927 07:02:01.795648 16774 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.796034 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.795898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44786)
0927 07:02:01.795949 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.796001 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:02:01.796195 16723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
I20260927 07:02:01.796679 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.796577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44792)
0927 07:02:01.796625 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.796656 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:02:01.796842 16774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:36427: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
I20260927 07:02:01.806784 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:36427
--webserver_interface=127.3.57.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:01.846084 16774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } attempt: 1
I20260927 07:02:01.846598 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.846452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44800)
0927 07:02:01.846516 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.846562 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:02:01.846951 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.846863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44802)
0927 07:02:01.846905 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.846932 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:02:01.847044 16774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:36427: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
I20260927 07:02:01.849215 16699 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } attempt: 3
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:02:01.856151 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.849492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57470)
0927 07:02:01.849546 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.849555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.849616 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.849764 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.849765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.849922 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.849983 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.849985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.850734 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.850736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.851236 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.851237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.851643 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.851644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.851727 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.852211 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.852216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.853561 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.855072 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.855073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.855073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.855253 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.855458 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.855459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.855460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.855525 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.855822 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.855823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.855870 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.855953 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.856053 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 07:02:01.856178 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.3.57.61:46637)
0927 07:02:01.849608 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.849610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.849620 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:02:01.849680 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.849681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.849690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.849760 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.850085 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.850634 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.851319 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.851532 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.851650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.853684 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.853686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.853686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.853694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.854984 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.855330 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.855331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.855331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.855389 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.855640 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.855641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.855641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.855752 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.855830 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.855992 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.856064 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:02:01.856607 16699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.856964 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.856857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44816)
0927 07:02:01.856911 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.856938 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:02:01.857254 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.857166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44826)
0927 07:02:01.857210 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.857231 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:02:01.857342 16699 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:36427: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
W20260927 07:02:01.888298 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:01.888526 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:01.888562 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 07:02:01.888505 16699 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } attempt: 1
I20260927 07:02:01.889021 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.888877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44842)
0927 07:02:01.888951 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.888989 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 07:02:01.889418 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.889317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44846)
0927 07:02:01.889363 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.889395 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:02:01.889501 16699 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:36427: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
W20260927 07:02:01.889956 16781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:01.889997 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:01.890010 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:01.890023 16781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:01.890086 16781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:01.891640 16781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40713,127.3.57.61:46637,127.3.57.60:36427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.57.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:01.891865 16781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:01.892105 16781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:01.896745 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:01.896786 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:01.896857 16781 server_base.cc:1034] running on GCE node
W20260927 07:02:01.896786 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:01.897215 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:01.897504 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:01.898680 16781 hybrid_clock.cc:648] HybridClock initialized: now 1790492521898659 us; error 45 us; skew 500 ppm
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:01.901234 16781 init.cc:377] Logged in from keytab as kudu/127.3.57.60@KRBTEST.COM (short username kudu)
I20260927 07:02:01.901657 16781 webserver.cc:460] Webserver started at http://127.3.57.60:33813/ using document root <none> and password file <none>
I20260927 07:02:01.901835 16781 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:01.901875 16781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:01.901957 16781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:01.902911 16781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data/instance:
uuid: "4d7689df65564ac6a375b0a9fb55b8da"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "a60bf6aa70e56d10e99e70c5cde549f3"
server_key_iv: "92fc63cef25dde32a65fe0c077dfdb7e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.903196 16781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal/instance:
uuid: "4d7689df65564ac6a375b0a9fb55b8da"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "a60bf6aa70e56d10e99e70c5cde549f3"
server_key_iv: "92fc63cef25dde32a65fe0c077dfdb7e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.904467 16781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:02:01.905160 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:01.905336 16781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:02:01.905422 16781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "4d7689df65564ac6a375b0a9fb55b8da"
format_stamp: "Formatted at 2026-09-27 07:02:01 on dist-test-slave-mk3h"
server_key: "a60bf6aa70e56d10e99e70c5cde549f3"
server_key_iv: "92fc63cef25dde32a65fe0c077dfdb7e"
server_key_version: "encryptionkey@0"
I20260927 07:02:01.905493 16781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:01.915180 16774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } attempt: 2
I20260927 07:02:01.915580 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.915430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44852)
0927 07:02:01.915485 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.915521 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:02:01.915969 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.915888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44864)
0927 07:02:01.915930 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.915954 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:02:01.916071 16774 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:36427: Network error: Client connection negotiation failed: client connection to 127.3.57.60:36427: connect: Connection refused (error 111)
I20260927 07:02:01.924530 16781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:01.926268 16781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:01.926369 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:01.930539 16781 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.60:36427
I20260927 07:02:01.930589 16848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.60:36427 every 8 connection(s)
I20260927 07:02:01.930945 16781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
I20260927 07:02:01.931264 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16781
I20260927 07:02:01.931372 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492341973781-3300-0/minicluster-data/master-2/wal/instance
I20260927 07:02:01.931612 3300 external_mini_cluster.cc:1428] Setting key 8c21dc805acf473ac3b45aefe7cf63d9
I20260927 07:02:01.931938 16849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:01.934232 16849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:01.941941 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.932242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.3.57.60:36427)
0927 07:02:01.932410 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.932419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.932441 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:02:01.933014 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.933020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.933064 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.933170 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.933934 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.935032 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.935859 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.936294 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.936399 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.938663 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.938671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.938671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.938678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.940423 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.940836 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.940838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.940839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.940912 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.941108 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.941109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.941109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.941317 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.941465 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.941565 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.941780 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260927 07:02:01.942459 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.934777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40713 (local address 127.0.0.1:47660)
0927 07:02:01.934973 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.935002 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.935059 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.935892 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.935895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.936135 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.936204 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.936205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.936894 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.936895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.937462 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.937464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.937840 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.937841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.937918 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.938377 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.938384 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.939695 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.941349 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.941355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.941357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.941625 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.941818 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.941819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.941819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.941885 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.942182 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.942186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.942245 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.942317 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.942391 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260927 07:02:01.942538 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.934847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.3.57.62:40713)
0927 07:02:01.934906 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.934907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.935072 (+ 165us) server_negotiation.cc:407] Connection header received
0927 07:02:01.935103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.935105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.935117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.935196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.936299 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.936803 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.937551 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.937744 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.937828 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.939778 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.939780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.939780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.939787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.941274 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.941687 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.941688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.941689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.941742 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.941955 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.941956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.941956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.942107 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.942176 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.942362 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.942443 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:02:01.943053 16849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:01.950105 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.943289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:46637 (local address 127.0.0.1:57486)
0927 07:02:01.943336 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.943344 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.943375 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.943487 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.943487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.943664 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.943699 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.943701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.944629 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.944630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.945382 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.945384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.945857 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.945858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.945912 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.946315 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.946334 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.947608 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.949104 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.949105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.949105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.949263 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.949471 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.949472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.949472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.949534 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.949776 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.949776 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.949816 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.949907 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.950007 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:02:01.950105 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.943346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.3.57.61:46637)
0927 07:02:01.943387 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.943388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.943396 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:01.943437 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.943437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.943445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.943481 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.943837 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.944549 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.945483 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.945780 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.945847 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.947684 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.947686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.947686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.947693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.949010 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.949333 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.949334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.949335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.949392 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.949596 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.949597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.949597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.949709 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.949783 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.949929 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.950005 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:02:01.950776 16849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:02:01.951629 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.942763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.3.57.62:40713)
0927 07:02:01.942799 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.942801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.943094 (+ 293us) server_negotiation.cc:407] Connection header received
0927 07:02:01.943151 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.943152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.943162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.943210 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.943839 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.944533 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.945483 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.945778 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.945847 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.948349 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.948350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.948350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.948358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.949758 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.950589 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.950590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.950590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.950653 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.950886 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.950887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.950888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.951017 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.951214 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.951455 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.951508 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:02:01.958039 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.950979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44884)
0927 07:02:01.951025 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.951034 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.951069 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.951183 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.951183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.951281 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.951303 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.951304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.952198 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.952199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.952933 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.952935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.953380 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.953381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.953414 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.953743 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.953748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.955199 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.956794 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.956795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.956795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.957021 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.957440 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.957441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.957441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.957494 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.957764 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.957765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.957809 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.957882 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.957976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 07:02:01.958168 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.951025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.3.57.60:36427)
0927 07:02:01.951061 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.951063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.951071 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:01.951114 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.951115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.951123 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.951182 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.951382 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.952131 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.953001 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.953312 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.953378 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.955271 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.955272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.955272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.955278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.956716 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.957086 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.957087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.957088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.957137 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.957562 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.957563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.957563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.957692 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.957756 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.957917 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.958050 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":19}
I20260927 07:02:01.959637 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.952129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.3.57.61:46637)
0927 07:02:01.952166 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.952168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.952176 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:01.952230 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.952231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.952239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.952295 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.952799 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.953412 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.954366 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.954671 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.954759 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.956992 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.956992 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.956993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.956998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.958636 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.958943 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.958943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.958944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.958996 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.959131 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.959131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.959131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.959246 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.959422 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.959463 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.959528 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:02:01.959735 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: Bootstrap starting.
I20260927 07:02:01.960731 16849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:01.961041 16849 log.cc:824] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:01.961378 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:02:01.961773 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: No bootstrap required, opened a new log
I20260927 07:02:01.963815 16849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:01.963959 16849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:01.964012 16849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7689df65564ac6a375b0a9fb55b8da, State: Initialized, Role: FOLLOWER
I20260927 07:02:01.964135 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:01.964432 16849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:01.964707 16699 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } attempt: 2
I20260927 07:02:01.965874 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:01.965955 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:02:01.966357 16781 master_runner.cc:186] Error getting master registration for 127.3.57.62:40713: OK, instance_id { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" instance_seqno: 1790492521623508 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:02:01.969389 16781 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:02:01.970386 16874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:02:01.970515 16874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 07:02:01 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:02:01.972041 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.962817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.3.57.60:36427)
0927 07:02:01.962883 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.962885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.962998 (+ 113us) server_negotiation.cc:407] Connection header received
0927 07:02:01.963089 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.963090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.963098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.963126 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.963891 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.964577 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.966465 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.966746 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.966798 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.968415 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.968416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.968416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.968422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.969912 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.970376 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.970377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.970377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.970653 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.971159 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.971161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.971161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.971322 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.971387 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.971889 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.971958 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:02:01.972992 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.963067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.3.57.61:46637)
0927 07:02:01.963098 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.963100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.963210 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:02:01.963300 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.963301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.963311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.963339 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.964034 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.964730 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.966648 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.966956 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.966995 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.969642 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.969643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.969643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.969650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.971374 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.971973 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.971974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.971975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.972027 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.972392 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.972394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.972394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.972517 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.972611 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.972822 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.972902 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:02:01.973471 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.962762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.3.57.62:40713)
0927 07:02:01.962856 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.962858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.962989 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:02:01.963087 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.963088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.963098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.963130 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.963956 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.964714 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.967301 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.967622 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.967662 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.969894 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.969895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.969895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.969901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.971670 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.972474 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.972475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.972475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.972534 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.972749 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.972750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.972750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.972863 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.972940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.973291 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.973373 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:02:01.974208 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.965073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.3.57.60:36427)
0927 07:02:01.965225 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:01.965226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:01.965234 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:01.965251 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:01.965252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:01.965260 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:01.965603 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:01.965658 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.966552 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.968192 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.968555 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.971110 (+ 2555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.971176 (+ 66us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:01.971177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:01.971177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:01.971182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:01.972920 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.973456 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.973457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.973457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.973515 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:01.973742 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:01.973743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:01.973743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:01.973853 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:01.973913 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:01.974046 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 07:02:01.974117 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260927 07:02:01.974220 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:01.965035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44902)
0927 07:02:01.965082 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:01.965091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:01.965132 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:01.965365 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:01.965366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:01.965513 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:01.965545 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.965547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.966646 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.966647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.967861 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:01.967864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:01.968920 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:01.968921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:01.968979 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:01.969569 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:01.969574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:01.971027 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:01.973036 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.973037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.973037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.973282 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.973612 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:01.973613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:01.973613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:01.973673 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:01.973908 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:01.973909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:01.973953 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:02:01.974020 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 07:02:01.974103 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 07:02:01.975342 16699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: Bootstrap starting.
I20260927 07:02:01.975997 16699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:01.976287 16699 log.cc:824] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:01.977013 16699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: No bootstrap required, opened a new log
I20260927 07:02:01.978440 16699 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:01.978577 16699 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:01.978618 16699 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9fc3e156a14478bdeafa6337fd64f5, State: Initialized, Role: FOLLOWER
I20260927 07:02:01.978739 16699 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:01.979039 16699 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:01.979126 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:01.979370 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:02:01.980883 16888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:02:01.980937 16888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:02:01.981500 16631 master_runner.cc:186] Error getting master registration for 127.3.57.61:46637: OK, instance_id { permanent_uuid: "e818826c0457497695417c1a44553d6e" instance_seqno: 1790492521773607 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:02:01.984261 16631 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:02:02.007225 16774 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } attempt: 3
Sep 27 07:02:02 dist-test-slave-mk3h krb5kdc[16618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:02:02.015836 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:02.007595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.3.57.60:36427)
0927 07:02:02.007646 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:02.007648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:02.007661 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:02:02.007682 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:02.007683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:02.007694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:02.007757 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:02.008127 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:02.008972 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:02.009990 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:02.010338 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:02.010409 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:02.012602 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:02.012603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:02.012604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:02.012610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:02.014337 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:02.014742 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:02.014744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:02.014744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:02.014805 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:02.015011 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:02.015012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:02.015012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:02.015164 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:02.015233 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:02.015403 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 07:02:02.015484 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 07:02:02.015594 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:02.007507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:36427 (local address 127.0.0.1:44914)
0927 07:02:02.007567 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:02.007578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:02.007617 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:02.007790 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:02.007791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:02.007980 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:02.008055 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:02.008057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:02.009054 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:02.009055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:02.009910 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:02.009913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:02.010417 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:02.010418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:02.010478 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:02.010977 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:02.010982 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:02.012513 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:02.014425 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:02.014427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:02.014427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:02.014667 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:02.014878 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:02.014878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:02.014879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:02.014945 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:02.015241 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:02.015242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:02.015290 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 07:02:02.015364 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:02:02.015465 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 07:02:02.017935 16774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: Bootstrap starting.
I20260927 07:02:02.018656 16774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:02.018981 16774 log.cc:824] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:02.019855 16774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: No bootstrap required, opened a new log
I20260927 07:02:02.021814 16774 raft_consensus.cc:348] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:02.021948 16774 raft_consensus.cc:374] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:02.021982 16774 raft_consensus.cc:729] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e818826c0457497695417c1a44553d6e, State: Initialized, Role: FOLLOWER
I20260927 07:02:02.022101 16774 consensus_queue.cc:260] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:02.022382 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.022428 16774 sys_catalog.cc:564] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:02.023617 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:02:02.024392 16902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:02:02.024451 16902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:02:02.026448 16706 master_runner.cc:263] No leader master found from master e818826c0457497695417c1a44553d6e
I20260927 07:02:02.026494 16706 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:02:02.121372 16652 tablet.cc:2369] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:02:02.177040 16802 tablet.cc:2369] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:02:02.212081 16862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:02.212206 16862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:02.212477 16862 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9fc3e156a14478bdeafa6337fd64f5 (127.3.57.62:40713), e818826c0457497695417c1a44553d6e (127.3.57.61:46637)
I20260927 07:02:02.212702 16673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7689df65564ac6a375b0a9fb55b8da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" is_pre_election: true
I20260927 07:02:02.212740 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7689df65564ac6a375b0a9fb55b8da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e818826c0457497695417c1a44553d6e" is_pre_election: true
I20260927 07:02:02.212863 16673 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7689df65564ac6a375b0a9fb55b8da in term 0.
I20260927 07:02:02.212872 16748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d7689df65564ac6a375b0a9fb55b8da in term 0.
I20260927 07:02:02.213066 16797 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7689df65564ac6a375b0a9fb55b8da, cf9fc3e156a14478bdeafa6337fd64f5; no voters:
I20260927 07:02:02.213228 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:02:02.213302 16862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:02:02.213337 16862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:02.214021 16862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:02.214159 16862 leader_election.cc:288] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [CANDIDATE]: Term 1 election: Requested vote from peers cf9fc3e156a14478bdeafa6337fd64f5 (127.3.57.62:40713), e818826c0457497695417c1a44553d6e (127.3.57.61:46637)
I20260927 07:02:02.214280 16748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7689df65564ac6a375b0a9fb55b8da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e818826c0457497695417c1a44553d6e"
I20260927 07:02:02.214365 16673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d7689df65564ac6a375b0a9fb55b8da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9fc3e156a14478bdeafa6337fd64f5"
I20260927 07:02:02.214386 16748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:02.214439 16673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:02.215189 16748 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7689df65564ac6a375b0a9fb55b8da in term 1.
I20260927 07:02:02.215276 16673 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d7689df65564ac6a375b0a9fb55b8da in term 1.
I20260927 07:02:02.215394 16799 leader_election.cc:302] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d7689df65564ac6a375b0a9fb55b8da, e818826c0457497695417c1a44553d6e; no voters:
I20260927 07:02:02.215478 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:02.215641 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [term 1 LEADER]: Becoming Leader. State: Replica: 4d7689df65564ac6a375b0a9fb55b8da, State: Running, Role: LEADER
I20260927 07:02:02.215737 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } }
I20260927 07:02:02.216238 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7689df65564ac6a375b0a9fb55b8da. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.216344 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:02.216531 16906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:02.216735 16906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:02:02.218094 16673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [term 1 FOLLOWER]: Refusing update from remote peer 4d7689df65564ac6a375b0a9fb55b8da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:02.218114 16748 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [term 1 FOLLOWER]: Refusing update from remote peer 4d7689df65564ac6a375b0a9fb55b8da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:02.218410 16904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:02.219304 16904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:02.219684 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7689df65564ac6a375b0a9fb55b8da. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.219764 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:02:02.220498 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d7689df65564ac6a375b0a9fb55b8da. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.220598 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:02:02.220616 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.220685 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:02.220830 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.220906 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:02:02.221277 16904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.221354 16904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:02.221359 16906 catalog_manager.cc:1347] Generated new cluster ID: 40915b13cce949188c26ce4a7043a7eb
I20260927 07:02:02.221446 16906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:02.221689 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d7689df65564ac6a375b0a9fb55b8da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9fc3e156a14478bdeafa6337fd64f5" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40713 } } peers { permanent_uuid: "e818826c0457497695417c1a44553d6e" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 46637 } } peers { permanent_uuid: "4d7689df65564ac6a375b0a9fb55b8da" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 36427 } } }
I20260927 07:02:02.221763 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:02:02.233379 16906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:02:02.233915 16906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:02.240547 16906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d7689df65564ac6a375b0a9fb55b8da: Generated new TSK 0
I20260927 07:02:02.240731 16906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:02:02.981375 16888 catalog_manager.cc:1259] Loaded cluster ID: 40915b13cce949188c26ce4a7043a7eb
I20260927 07:02:02.981469 16888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: loading cluster ID for follower catalog manager: success
I20260927 07:02:02.982311 16888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: acquiring CA information for follower catalog manager: success
I20260927 07:02:02.982611 16888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf9fc3e156a14478bdeafa6337fd64f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:02:03.024855 16902 catalog_manager.cc:1259] Loaded cluster ID: 40915b13cce949188c26ce4a7043a7eb
I20260927 07:02:03.024950 16902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: loading cluster ID for follower catalog manager: success
I20260927 07:02:03.025729 16902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: acquiring CA information for follower catalog manager: success
I20260927 07:02:03.026036 16902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e818826c0457497695417c1a44553d6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:02:03.296362 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16631
I20260927 07:02:03.301071 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16706
I20260927 07:02:03.306007 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16781
2026-09-27T07:02:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2953 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:02:03 dist-test-slave-mk3h krb5kdc[16921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:02:03 dist-test-slave-mk3h krb5kdc[16921](info): set up 2 sockets
Sep 27 07:02:03 dist-test-slave-mk3h krb5kdc[16921](info): commencing operation
krb5kdc: starting...
W20260927 07:02:04.347656 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[16921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:02:04.458124 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36887
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:04.535784 16934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:04.536007 16934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:04.536043 16934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:04.537189 16934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:04.537266 16934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:04.537293 16934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:04.537323 16934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:04.537348 16934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:04.538786 16934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:04.539047 16934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:04.539292 16934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:04.543941 16940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:04.544085 16934 server_base.cc:1034] running on GCE node
W20260927 07:02:04.543958 16942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:04.544230 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:04.544467 16934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:04.544790 16934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:04.545926 16934 hybrid_clock.cc:648] HybridClock initialized: now 1790492524545915 us; error 29 us; skew 500 ppm
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[16921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:04.548375 16934 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:04.548808 16934 webserver.cc:460] Webserver started at http://127.3.57.62:37379/ using document root <none> and password file <none>
I20260927 07:02:04.549014 16934 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:04.549053 16934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:04.549136 16934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:04.550105 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "9f159d6e194b44f0a5ebc95824c93e12"
format_stamp: "Formatted at 2026-09-27 07:02:04 on dist-test-slave-mk3h"
server_key: "99d18b7a9becc9424ce36170428138fb"
server_key_iv: "af7d2a3968625e447714cadfbba67150"
server_key_version: "encryptionkey@0"
I20260927 07:02:04.550405 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "9f159d6e194b44f0a5ebc95824c93e12"
format_stamp: "Formatted at 2026-09-27 07:02:04 on dist-test-slave-mk3h"
server_key: "99d18b7a9becc9424ce36170428138fb"
server_key_iv: "af7d2a3968625e447714cadfbba67150"
server_key_version: "encryptionkey@0"
I20260927 07:02:04.551632 16934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:02:04.552348 16949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:04.552516 16934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:04.552618 16934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "9f159d6e194b44f0a5ebc95824c93e12"
format_stamp: "Formatted at 2026-09-27 07:02:04 on dist-test-slave-mk3h"
server_key: "99d18b7a9becc9424ce36170428138fb"
server_key_iv: "af7d2a3968625e447714cadfbba67150"
server_key_version: "encryptionkey@0"
I20260927 07:02:04.552688 16934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:04.572299 16934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:04.574169 16934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:04.574323 16934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:04.578265 16934 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:36887
I20260927 07:02:04.578296 17001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:36887 every 8 connection(s)
I20260927 07:02:04.578630 16934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:04.579424 17002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:04.582140 17002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: Bootstrap starting.
I20260927 07:02:04.582247 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 16934
I20260927 07:02:04.582350 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:02:04.582542 3300 external_mini_cluster.cc:1428] Setting key b3fba150b1c6e36866c94b5a68ab12d1
I20260927 07:02:04.582808 17002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:04.583144 17002 log.cc:824] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:04.583879 17002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: No bootstrap required, opened a new log
I20260927 07:02:04.586124 17002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } }
I20260927 07:02:04.586284 17002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:04.586318 17002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f159d6e194b44f0a5ebc95824c93e12, State: Initialized, Role: FOLLOWER
I20260927 07:02:04.586436 17002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } }
I20260927 07:02:04.586503 17002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:04.586530 17002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:04.586570 17002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:04.587137 17002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } }
I20260927 07:02:04.587220 17002 leader_election.cc:302] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f159d6e194b44f0a5ebc95824c93e12; no voters:
I20260927 07:02:04.587426 17002 leader_election.cc:288] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:02:04.587448 17007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:04.587577 17007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [term 1 LEADER]: Becoming Leader. State: Replica: 9f159d6e194b44f0a5ebc95824c93e12, State: Running, Role: LEADER
I20260927 07:02:04.587659 17007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } }
I20260927 07:02:04.587698 17002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:04.587999 17008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f159d6e194b44f0a5ebc95824c93e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } } }
I20260927 07:02:04.588253 17008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:04.588160 17009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f159d6e194b44f0a5ebc95824c93e12. Latest consensus state: current_term: 1 leader_uuid: "9f159d6e194b44f0a5ebc95824c93e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f159d6e194b44f0a5ebc95824c93e12" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36887 } } }
I20260927 07:02:04.589370 17009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12 [sys.catalog]: This master's current role is: LEADER
W20260927 07:02:04.589376 17022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:02:04.589440 17022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:02:04.589515 17023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:04.589721 17023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[16921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:04.591470 17023 catalog_manager.cc:1347] Generated new cluster ID: 736ec1bfc41e43fb85bde84338b02288
I20260927 07:02:04.591521 17023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:04.598160 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:04.583017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.3.57.62:36887)
0927 07:02:04.583206 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:04.583215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:04.583246 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:02:04.583814 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:04.583823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:04.583890 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:04.584035 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:04.584903 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:04.585759 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:04.586801 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:04.587208 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:04.587333 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:04.590583 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:04.590586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:04.590587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:04.590597 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:04.596553 (+ 5956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:04.596967 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:04.596970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:04.596970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:04.597081 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:04.597292 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:04.597293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:04.597294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:04.597569 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:04.597705 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:04.597832 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 07:02:04.597952 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260927 07:02:04.598973 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:02:04.619890 17023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:02:04.620420 17023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:04.623090 17023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f159d6e194b44f0a5ebc95824c93e12: Generated new TSK 0
I20260927 07:02:04.623265 17023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:02:04.652832 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 16934
2026-09-27T07:02:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1344 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[17029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[17029](info): set up 2 sockets
Sep 27 07:02:04 dist-test-slave-mk3h krb5kdc[17029](info): commencing operation
krb5kdc: starting...
W20260927 07:02:05.683718 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:02:05 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:02:05.795460 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:05.871384 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:05.871624 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:05.871659 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:05.872761 17042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:05.872828 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:05.872852 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:05.872879 17042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:05.872918 17042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:05.874308 17042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:05.874573 17042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:05.874778 17042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:05.879351 17048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:05.879438 17047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:05.879340 17050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:05.879444 17042 server_base.cc:1034] running on GCE node
I20260927 07:02:05.879774 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:05.880054 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:05.881213 17042 hybrid_clock.cc:648] HybridClock initialized: now 1790492525881193 us; error 37 us; skew 500 ppm
Sep 27 07:02:05 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:05.883579 17042 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:05.883991 17042 webserver.cc:460] Webserver started at http://127.3.57.62:44739/ using document root <none> and password file <none>
I20260927 07:02:05.884193 17042 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:05.884255 17042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:05.884397 17042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:05.885358 17042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "2e1fa46a16d94d5db674ad47ed826c6b"
format_stamp: "Formatted at 2026-09-27 07:02:05 on dist-test-slave-mk3h"
server_key: "c5c19261734aa05defe0f0b4c7aeffc3"
server_key_iv: "a8032d3ed44132be1ab4d8d5109e9d22"
server_key_version: "encryptionkey@0"
I20260927 07:02:05.885689 17042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "2e1fa46a16d94d5db674ad47ed826c6b"
format_stamp: "Formatted at 2026-09-27 07:02:05 on dist-test-slave-mk3h"
server_key: "c5c19261734aa05defe0f0b4c7aeffc3"
server_key_iv: "a8032d3ed44132be1ab4d8d5109e9d22"
server_key_version: "encryptionkey@0"
I20260927 07:02:05.886926 17042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:02:05.887610 17057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:05.887766 17042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:05.887867 17042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "2e1fa46a16d94d5db674ad47ed826c6b"
format_stamp: "Formatted at 2026-09-27 07:02:05 on dist-test-slave-mk3h"
server_key: "c5c19261734aa05defe0f0b4c7aeffc3"
server_key_iv: "a8032d3ed44132be1ab4d8d5109e9d22"
server_key_version: "encryptionkey@0"
I20260927 07:02:05.887949 17042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:05.912158 17042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:05.914417 17042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:05.914575 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:05.918361 17042 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35101
I20260927 07:02:05.918417 17109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35101 every 8 connection(s)
I20260927 07:02:05.918757 17042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:05.919418 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17042
I20260927 07:02:05.919531 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:02:05.919692 17110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:05.919751 3300 external_mini_cluster.cc:1428] Setting key efebb84b59608a77c5cada9eed84d5e9
I20260927 07:02:05.922657 17110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap starting.
I20260927 07:02:05.923336 17110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:05.923645 17110 log.cc:824] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:05.924290 17110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: No bootstrap required, opened a new log
Sep 27 07:02:05 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:05.926688 17110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:05.926841 17110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:05.926879 17110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Initialized, Role: FOLLOWER
I20260927 07:02:05.927026 17110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:05.927132 17110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:05.927158 17110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:05.927222 17110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:05.927757 17110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:05.927862 17110 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1fa46a16d94d5db674ad47ed826c6b; no voters:
I20260927 07:02:05.928134 17110 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:02:05.928162 17115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:05.928273 17115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 LEADER]: Becoming Leader. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Running, Role: LEADER
I20260927 07:02:05.928390 17110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:05.928408 17115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:05.928710 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1fa46a16d94d5db674ad47ed826c6b. Latest consensus state: current_term: 1 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:05.928809 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:05.928725 17116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:05.928954 17116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:05.929013 17121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:05.929507 17121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:02:05.930684 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:05.920249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.3.57.62:35101)
0927 07:02:05.920409 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:05.920416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:05.920439 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:02:05.921019 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:05.921025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:05.921068 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:05.921174 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:05.921969 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:05.922897 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:05.923959 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:05.924420 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:05.924509 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:05.926846 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:05.926850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:05.926850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:05.926862 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:05.928896 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:05.929299 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:05.929302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:05.929302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:05.929410 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:05.929672 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:05.929673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:05.929674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:05.929978 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:05.930132 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:05.930267 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 07:02:05.930493 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260927 07:02:05.931082 17121 catalog_manager.cc:1347] Generated new cluster ID: f3fa951941ec4d6a8109635cbd56a29e
I20260927 07:02:05.931134 17121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:05.938030 17121 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:02:05.938670 17121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:05.949703 17121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Generated new TSK 0
I20260927 07:02:05.949847 17121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:02:05.975129 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:06.053879 17136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.054111 17136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.054147 17136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.055294 17136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.055356 17136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.055397 17136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:02:06.056852 17136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.057104 17136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.057327 17136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.062247 17141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.062276 17136 server_base.cc:1034] running on GCE node
W20260927 07:02:06.062271 17142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.062383 17144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.062665 17136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.062865 17136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.063998 17136 hybrid_clock.cc:648] HybridClock initialized: now 1790492526063974 us; error 36 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.066308 17136 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:02:06.066704 17136 webserver.cc:460] Webserver started at http://127.3.57.1:36965/ using document root <none> and password file <none>
I20260927 07:02:06.066933 17136 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.067011 17136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.067144 17136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:06.068135 17136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "a2687d43e81f430d98403242a40856ee"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "de2acc551b0a818cf68afd92dc4a4391"
server_key_iv: "a98c0814fa883df07c3c3755495c1ed9"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.068471 17136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "a2687d43e81f430d98403242a40856ee"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "de2acc551b0a818cf68afd92dc4a4391"
server_key_iv: "a98c0814fa883df07c3c3755495c1ed9"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.069790 17136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:02:06.070488 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.070649 17136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:06.070745 17136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "a2687d43e81f430d98403242a40856ee"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "de2acc551b0a818cf68afd92dc4a4391"
server_key_iv: "a98c0814fa883df07c3c3755495c1ed9"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.070819 17136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.082118 17136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.083827 17136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.083971 17136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.084175 17136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:06.084462 17136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:06.084513 17136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.084545 17136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:06.084573 17136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.090023 17136 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:40803
I20260927 07:02:06.090090 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:40803 every 8 connection(s)
I20260927 07:02:06.090427 17136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:06.098407 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.091093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:44141 (local address 127.3.57.62:35101)
0927 07:02:06.091139 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.091141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.091347 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:02:06.091866 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.091866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.091876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.091926 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.092721 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.093197 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.094080 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.094262 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.094308 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.096209 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.096210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.096210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.096216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.097351 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.097674 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.097675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.097675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.097720 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.097929 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.097929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.097930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.098021 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.098068 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.098269 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.098336 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:02:06.098551 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.091084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:44141)
0927 07:02:06.091236 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.091267 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.091785 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.091973 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.091976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.092482 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:06.092654 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.092657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.093270 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.093271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.094014 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.094016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.094336 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.094337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.094424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.094878 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:06.094885 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:06.096128 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:06.097418 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.097423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.097425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.097614 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.097792 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.097792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.097793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.097878 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.098072 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:06.098075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:06.098171 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.098243 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.098358 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260927 07:02:06.099020 17265 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:06.099113 17265 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:06.099278 17265 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:06.099691 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17136
I20260927 07:02:06.099783 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:02:06.099887 17074 ts_manager.cc:194] Registered new tserver with Master: a2687d43e81f430d98403242a40856ee (127.3.57.1:40803)
I20260927 07:02:06.099997 3300 external_mini_cluster.cc:1428] Setting key f400e67f3120aba6dca0d7b8f66069bb
I20260927 07:02:06.100651 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:44141
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:02:06.122438 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:06.212297 17270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.212598 17270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.212641 17270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.214738 17270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.214794 17270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.214840 17270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:02:06.217509 17270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.217779 17270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.218057 17270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.223043 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.223052 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.223054 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.223273 17270 server_base.cc:1034] running on GCE node
I20260927 07:02:06.223438 17270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.223680 17270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.224812 17270 hybrid_clock.cc:648] HybridClock initialized: now 1790492526224793 us; error 32 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.227133 17270 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:02:06.227624 17270 webserver.cc:460] Webserver started at http://127.3.57.2:33755/ using document root <none> and password file <none>
I20260927 07:02:06.227842 17270 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.227907 17270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.228060 17270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:06.229106 17270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "45931ab91b2f41d9af4768311ee9263c"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "589a364392832649d7f7b96d0c61dc34"
server_key_iv: "e52b3a8d9eb144ac431e7034685a21e1"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.229476 17270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "45931ab91b2f41d9af4768311ee9263c"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "589a364392832649d7f7b96d0c61dc34"
server_key_iv: "e52b3a8d9eb144ac431e7034685a21e1"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.230782 17270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:02:06.231458 17285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.231643 17270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:02:06.231741 17270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "45931ab91b2f41d9af4768311ee9263c"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "589a364392832649d7f7b96d0c61dc34"
server_key_iv: "e52b3a8d9eb144ac431e7034685a21e1"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.231815 17270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.247627 17270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.249408 17270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.249560 17270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.249776 17270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:06.250079 17270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:06.250137 17270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.250177 17270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:06.250244 17270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.255897 17270 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:40677
I20260927 07:02:06.255939 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:40677 every 8 connection(s)
I20260927 07:02:06.256251 17270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:02:06.256464 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17270
I20260927 07:02:06.256563 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:02:06.256764 3300 external_mini_cluster.cc:1428] Setting key 72b01c69b8a90c63fddd9347264bf61e
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:02:06.266649 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.256955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:35155 (local address 127.3.57.62:35101)
0927 07:02:06.257017 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.257019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.257287 (+ 268us) server_negotiation.cc:407] Connection header received
0927 07:02:06.257763 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.257764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.257777 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.257827 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.258796 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.259292 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.260841 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.261113 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.261194 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.263790 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.263792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.263792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.263800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.265418 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.265854 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.265855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.265856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.265907 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.266114 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.266115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.266115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.266209 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.266257 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.266523 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.266587 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 07:02:06.266789 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.256930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:35155)
0927 07:02:06.257162 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.257200 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.257650 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.257900 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.257904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.258438 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:06.258675 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.258682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.259409 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.259410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.260759 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.260762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.261212 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.261213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.261360 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.261860 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:06.261871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:06.263694 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:06.265513 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.265520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.265522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.265791 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.265961 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.265962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.265962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.266062 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.266265 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:06.266269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:06.266416 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.266494 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.266598 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":168,"threads_started":1}
I20260927 07:02:06.267390 17399 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:06.267490 17399 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
I20260927 07:02:06.267674 17399 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:06.268086 17074 ts_manager.cc:194] Registered new tserver with Master: 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677)
I20260927 07:02:06.268455 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:35155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:02:06.280309 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:06.358343 17404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.358530 17404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.358553 17404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.359680 17404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.359717 17404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.359767 17404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:02:06.361176 17404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.361416 17404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.361634 17404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.366719 17409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.366801 17410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.366717 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.366832 17404 server_base.cc:1034] running on GCE node
I20260927 07:02:06.367121 17404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.367339 17404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.368482 17404 hybrid_clock.cc:648] HybridClock initialized: now 1790492526368478 us; error 39 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.370829 17404 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:02:06.371273 17404 webserver.cc:460] Webserver started at http://127.3.57.3:38473/ using document root <none> and password file <none>
I20260927 07:02:06.371506 17404 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.371592 17404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.371744 17404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:06.372690 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "8269c02d3360426d80d917330316aea8"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "ce25383ec5a749518bc2bfbc0e266456"
server_key_iv: "893a5808efc5831980e40c27ebf5775a"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.373055 17404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "8269c02d3360426d80d917330316aea8"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "ce25383ec5a749518bc2bfbc0e266456"
server_key_iv: "893a5808efc5831980e40c27ebf5775a"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.374437 17404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:02:06.375175 17419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.375382 17404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:06.375479 17404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "8269c02d3360426d80d917330316aea8"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "ce25383ec5a749518bc2bfbc0e266456"
server_key_iv: "893a5808efc5831980e40c27ebf5775a"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.375588 17404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.391315 17404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.393028 17404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.393177 17404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.393391 17404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:06.393677 17404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:06.393709 17404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.393729 17404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:06.393772 17404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.399360 17404 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:40277
I20260927 07:02:06.399406 17532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:40277 every 8 connection(s)
I20260927 07:02:06.399765 17404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:02:06.404531 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17404
I20260927 07:02:06.404634 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:02:06.404822 3300 external_mini_cluster.cc:1428] Setting key e40f1214ef8d637ba1e89596240c4e7c
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:06.408977 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.400383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:37871 (local address 127.3.57.62:35101)
0927 07:02:06.400464 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.400466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.400712 (+ 246us) server_negotiation.cc:407] Connection header received
0927 07:02:06.401148 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.401149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.401158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.401204 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.402136 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.402646 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.403730 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.403939 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.404027 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.406448 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.406449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.406449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.406457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.407639 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.408022 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.408023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.408024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.408084 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.408328 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.408329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.408329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.408448 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.408530 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.408786 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.408883 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:02:06.409183 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.400365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:37871)
0927 07:02:06.400597 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.400630 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.401070 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.401262 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.401265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.401811 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:06.401989 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.401993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.402761 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.402762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.403636 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.403640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.404036 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.404037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.404169 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.404794 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:06.404803 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:06.406359 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:06.407723 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.407729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.407731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.407946 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.408152 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.408153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.408153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.408261 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.408531 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:06.408537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:06.408664 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.408764 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.408942 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
I20260927 07:02:06.409646 17533 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:06.409760 17533 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:06.409932 17533 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:06.410287 17074 ts_manager.cc:194] Registered new tserver with Master: 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
I20260927 07:02:06.410619 17074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:37871
I20260927 07:02:06.417161 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:02:06.423802 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.418529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.3.57.62:35101)
0927 07:02:06.418563 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.418564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.418689 (+ 125us) server_negotiation.cc:407] Connection header received
0927 07:02:06.418738 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.418738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.418746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.418788 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.419270 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.419781 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.420498 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.420665 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.420712 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.421833 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.421833 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.421834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.421838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.422911 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.423192 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.423193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.423193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.423237 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.423350 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.423350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.423350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.423462 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.423531 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.423690 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.423746 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:02:06.425388 17074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:02:06.426887 17074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:02:06.433638 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40277 (local address 127.0.0.1:34468)
0927 07:02:06.429479 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.429488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.429529 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.430060 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.430061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.430063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.430137 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.430139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431416 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.431417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.432354 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.432356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.433283 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.433284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.433393 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.433410 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.433503 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.433566 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":39,"threads_started":1}
I20260927 07:02:06.433645 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.3.57.3:40277)
0927 07:02:06.429855 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.429859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.429869 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:02:06.429903 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.429905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.429919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.429991 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.430216 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431305 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.432459 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.433166 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.433256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.433314 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.433527 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.433567 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":72,"threads_started":1}
I20260927 07:02:06.433725 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.3.57.2:40677)
0927 07:02:06.429548 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.429553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.429564 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:02:06.429604 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.429605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.429620 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.429723 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.430023 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431162 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.432617 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.433305 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.433372 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.433421 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.433568 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.433615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260927 07:02:06.434131 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:40677 (local address 127.0.0.1:58474)
0927 07:02:06.429185 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.429222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.429286 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.429727 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.429730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.429732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.429917 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.429920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431258 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.431259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.432532 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.432534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.433391 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.433392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.433445 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.433448 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.434011 (+ 563us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.434058 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":37,"thread_start_us":47,"threads_started":1}
I20260927 07:02:06.434418 17467 tablet_service.cc:1467] Processing CreateTablet for tablet 6e27a473828d417f8a916ea636fdc57c (DEFAULT_TABLE table=test-table [id=01c3af1e8ebb4023aa6de354ef72fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:06.434798 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e27a473828d417f8a916ea636fdc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:06.434907 17333 tablet_service.cc:1467] Processing CreateTablet for tablet 6e27a473828d417f8a916ea636fdc57c (DEFAULT_TABLE table=test-table [id=01c3af1e8ebb4023aa6de354ef72fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:06.435210 17333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e27a473828d417f8a916ea636fdc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:06.436198 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40803 (local address 127.0.0.1:38630)
0927 07:02:06.429416 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.429427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.429480 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.430019 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.430020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.430021 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.430071 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.430072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431148 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.431149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.432532 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.432534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.435953 (+ 3419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.435954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.436021 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.436025 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.436084 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.436129 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":66,"thread_start_us":61,"threads_started":1}
I20260927 07:02:06.436218 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.429244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.3.57.1:40803)
0927 07:02:06.429784 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.429792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.429802 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:02:06.429845 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.429847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.429862 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.429934 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.430158 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.431047 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.433455 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.435845 (+ 2390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.435949 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.436007 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.436099 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.436140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":83,"threads_started":1}
I20260927 07:02:06.438151 17547 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Bootstrap starting.
I20260927 07:02:06.438593 17548 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Bootstrap starting.
I20260927 07:02:06.438866 17547 tablet_bootstrap.cc:654] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:06.438761 17199 tablet_service.cc:1467] Processing CreateTablet for tablet 6e27a473828d417f8a916ea636fdc57c (DEFAULT_TABLE table=test-table [id=01c3af1e8ebb4023aa6de354ef72fa8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:06.439109 17199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e27a473828d417f8a916ea636fdc57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:06.439131 17548 tablet_bootstrap.cc:654] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:06.439421 17547 log.cc:824] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:06.439586 17548 log.cc:824] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:06.440238 17547 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: No bootstrap required, opened a new log
I20260927 07:02:06.440312 17547 ts_tablet_manager.cc:1397] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:02:06.440333 17548 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: No bootstrap required, opened a new log
I20260927 07:02:06.440394 17548 ts_tablet_manager.cc:1397] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:02:06.442049 17551 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Bootstrap starting.
I20260927 07:02:06.442205 17547 raft_consensus.cc:348] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.442205 17548 raft_consensus.cc:348] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.442381 17547 raft_consensus.cc:374] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:06.442382 17548 raft_consensus.cc:374] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:06.442420 17548 raft_consensus.cc:729] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45931ab91b2f41d9af4768311ee9263c, State: Initialized, Role: FOLLOWER
I20260927 07:02:06.442420 17547 raft_consensus.cc:729] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8269c02d3360426d80d917330316aea8, State: Initialized, Role: FOLLOWER
I20260927 07:02:06.442534 17547 consensus_queue.cc:260] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.442560 17548 consensus_queue.cc:260] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.442669 17551 tablet_bootstrap.cc:654] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:06.442811 17548 ts_tablet_manager.cc:1428] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:02:06.442811 17547 ts_tablet_manager.cc:1428] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 07:02:06.443063 17533 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
I20260927 07:02:06.443046 17399 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
I20260927 07:02:06.443255 17551 log.cc:824] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:06.444114 17551 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: No bootstrap required, opened a new log
I20260927 07:02:06.444186 17551 ts_tablet_manager.cc:1397] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:02:06.445688 17551 raft_consensus.cc:348] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.445809 17551 raft_consensus.cc:374] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:06.445833 17551 raft_consensus.cc:729] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2687d43e81f430d98403242a40856ee, State: Initialized, Role: FOLLOWER
I20260927 07:02:06.445902 17551 consensus_queue.cc:260] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.446097 17551 ts_tablet_manager.cc:1428] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:02:06.446163 17265 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
W20260927 07:02:06.507395 17400 tablet.cc:2369] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:02:06.526458 17555 raft_consensus.cc:482] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:06.526610 17555 raft_consensus.cc:504] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.527006 17555 leader_election.cc:288] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
I20260927 07:02:06.530423 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.527208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:40677 (local address 127.3.57.1:33967)
0927 07:02:06.527452 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.527463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.527521 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.527733 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.527734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.527736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.527826 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.527828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.528515 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.528516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.529442 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.529444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.530147 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.530148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.530238 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.530242 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.530284 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.530314 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":66,"thread_start_us":119,"threads_started":1}
I20260927 07:02:06.530426 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.527310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:33967 (local address 127.3.57.2:40677)
0927 07:02:06.527354 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.527356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.527535 (+ 179us) server_negotiation.cc:407] Connection header received
0927 07:02:06.527646 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.527647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.527661 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.527718 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.527919 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.528440 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.529523 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.530066 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.530122 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.530131 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.530310 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.530355 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:02:06.530503 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.527296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:47685 (local address 127.3.57.3:40277)
0927 07:02:06.527344 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.527346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.527358 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:02:06.527388 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.527389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.527402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.527482 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.527717 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.528333 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.529587 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.530084 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.530174 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.530183 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.530393 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.530433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:02:06.530529 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.527208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40277 (local address 127.3.57.1:47685)
0927 07:02:06.527274 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.527287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.527330 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.527491 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.527492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.527494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.527620 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.527623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.528408 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.528409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.529471 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.529474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.530210 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.530211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.530311 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.530315 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.530370 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.530408 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":32}
I20260927 07:02:06.530972 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e27a473828d417f8a916ea636fdc57c" candidate_uuid: "a2687d43e81f430d98403242a40856ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45931ab91b2f41d9af4768311ee9263c" is_pre_election: true
I20260927 07:02:06.530985 17487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e27a473828d417f8a916ea636fdc57c" candidate_uuid: "a2687d43e81f430d98403242a40856ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8269c02d3360426d80d917330316aea8" is_pre_election: true
I20260927 07:02:06.531091 17353 raft_consensus.cc:2454] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2687d43e81f430d98403242a40856ee in term 0.
I20260927 07:02:06.531126 17487 raft_consensus.cc:2454] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2687d43e81f430d98403242a40856ee in term 0.
I20260927 07:02:06.531325 17154 leader_election.cc:302] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8269c02d3360426d80d917330316aea8, a2687d43e81f430d98403242a40856ee; no voters:
I20260927 07:02:06.531461 17555 raft_consensus.cc:2781] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:02:06.531524 17555 raft_consensus.cc:482] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:02:06.531574 17555 raft_consensus.cc:3037] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:06.532227 17555 raft_consensus.cc:504] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.532363 17555 leader_election.cc:288] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 1 election: Requested vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
I20260927 07:02:06.532506 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e27a473828d417f8a916ea636fdc57c" candidate_uuid: "a2687d43e81f430d98403242a40856ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45931ab91b2f41d9af4768311ee9263c"
I20260927 07:02:06.532518 17487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e27a473828d417f8a916ea636fdc57c" candidate_uuid: "a2687d43e81f430d98403242a40856ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8269c02d3360426d80d917330316aea8"
I20260927 07:02:06.532583 17353 raft_consensus.cc:3037] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:06.532615 17487 raft_consensus.cc:3037] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:06.533238 17353 raft_consensus.cc:2454] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2687d43e81f430d98403242a40856ee in term 1.
I20260927 07:02:06.533420 17152 leader_election.cc:302] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45931ab91b2f41d9af4768311ee9263c, a2687d43e81f430d98403242a40856ee; no voters:
I20260927 07:02:06.533421 17487 raft_consensus.cc:2454] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2687d43e81f430d98403242a40856ee in term 1.
I20260927 07:02:06.533505 17555 raft_consensus.cc:2781] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:06.533666 17555 raft_consensus.cc:686] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 LEADER]: Becoming Leader. State: Replica: a2687d43e81f430d98403242a40856ee, State: Running, Role: LEADER
I20260927 07:02:06.533754 17555 consensus_queue.cc:237] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.534617 17074 catalog_manager.cc:5495] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee reported cstate change: term changed from 0 to 1, leader changed from <none> to a2687d43e81f430d98403242a40856ee (127.3.57.1). New cstate: current_term: 1 leader_uuid: "a2687d43e81f430d98403242a40856ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } health_report { overall_health: UNKNOWN } } }
I20260927 07:02:06.547717 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.545360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.3.57.1:40803)
0927 07:02:06.545403 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.545405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.545414 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:02:06.545496 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.545497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.545515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.545600 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:06.545723 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.546171 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.546911 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.547123 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.547196 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.547363 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:06.547457 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:02:06.547540 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.547606 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.547624 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:02:06.549288 17353 raft_consensus.cc:1261] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Refusing update from remote peer a2687d43e81f430d98403242a40856ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:06.549295 17487 raft_consensus.cc:1261] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Refusing update from remote peer a2687d43e81f430d98403242a40856ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:06.549603 17555 consensus_queue.cc:1035] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:06.549741 17557 consensus_queue.cc:1035] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:06.552448 17561 mvcc.cc:205] Tried to move back new op lower bound from 7333857388743368704 to 7333857388682633216. Current Snapshot: MvccSnapshot[applied={T|T < 7333857388743368704}]
I20260927 07:02:06.552634 17560 mvcc.cc:205] Tried to move back new op lower bound from 7333857388743368704 to 7333857388682633216. Current Snapshot: MvccSnapshot[applied={T|T < 7333857388743368704 or (T in {7333857388743368704})}]
I20260927 07:02:06.553771 17562 mvcc.cc:205] Tried to move back new op lower bound from 7333857388743368704 to 7333857388682633216. Current Snapshot: MvccSnapshot[applied={T|T < 7333857388743368704}]
I20260927 07:02:06.553937 17073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:06.554032 17073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:06.554749 17073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Sending DeleteTablet for 3 replicas of tablet 6e27a473828d417f8a916ea636fdc57c
I20260927 07:02:06.555020 17333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e27a473828d417f8a916ea636fdc57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:06 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:58474
I20260927 07:02:06.555078 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17136
I20260927 07:02:06.555083 17467 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e27a473828d417f8a916ea636fdc57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:06 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:34468
I20260927 07:02:06.555207 17199 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e27a473828d417f8a916ea636fdc57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:06 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:38630
I20260927 07:02:06.555714 17571 tablet_replica.cc:330] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: stopping tablet replica
I20260927 07:02:06.555826 17571 raft_consensus.cc:2229] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:06.555927 17571 raft_consensus.cc:2258] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:06.556384 17571 ts_tablet_manager.cc:1905] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:06.558001 17571 ts_tablet_manager.cc:1918] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:06.558116 17571 log.cc:1196] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/6e27a473828d417f8a916ea636fdc57c
I20260927 07:02:06.558352 17571 ts_tablet_manager.cc:1939] T 6e27a473828d417f8a916ea636fdc57c P 8269c02d3360426d80d917330316aea8: Deleting consensus metadata
I20260927 07:02:06.558749 17060 catalog_manager.cc:4841] TS 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): tablet 6e27a473828d417f8a916ea636fdc57c (table test-table [id=01c3af1e8ebb4023aa6de354ef72fa8c]) successfully deleted
I20260927 07:02:06.559664 17570 tablet_replica.cc:330] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: stopping tablet replica
I20260927 07:02:06.559765 17570 raft_consensus.cc:2229] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:06.559856 17570 raft_consensus.cc:2258] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:06.560217 17570 ts_tablet_manager.cc:1905] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 07:02:06.561275 17060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:40803 (error 108)
I20260927 07:02:06.561452 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17270
I20260927 07:02:06.561538 17570 ts_tablet_manager.cc:1918] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:06.561595 17570 log.cc:1196] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/6e27a473828d417f8a916ea636fdc57c
I20260927 07:02:06.561800 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.561683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40803 (local address 127.0.0.1:38638)
0927 07:02:06.561735 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.561764 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 07:02:06.561811 17570 ts_tablet_manager.cc:1939] T 6e27a473828d417f8a916ea636fdc57c P 45931ab91b2f41d9af4768311ee9263c: Deleting consensus metadata
W20260927 07:02:06.561898 17060 catalog_manager.cc:4584] TS a2687d43e81f430d98403242a40856ee (127.3.57.1:40803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e27a473828d417f8a916ea636fdc57c: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40803: connect: Connection refused (error 111)
I20260927 07:02:06.566702 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17404
I20260927 07:02:06.566758 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.566638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:40677 (local address 127.0.0.1:58488)
0927 07:02:06.566687 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.566712 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:40677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 07:02:06.570852 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17042
I20260927 07:02:06.576011 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--webserver_interface=127.3.57.62
--webserver_port=44739
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:06.653942 17573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.654158 17573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.654196 17573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.655300 17573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:06.655364 17573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.655390 17573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.655426 17573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:06.655475 17573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:06.656912 17573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=44739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.657177 17573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.657388 17573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.665653 17582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.665709 17579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.665655 17580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.665886 17573 server_base.cc:1034] running on GCE node
I20260927 07:02:06.666150 17573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.666477 17573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.667620 17573 hybrid_clock.cc:648] HybridClock initialized: now 1790492526667593 us; error 31 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.676699 17573 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:06.677148 17573 webserver.cc:460] Webserver started at http://127.3.57.62:44739/ using document root <none> and password file <none>
I20260927 07:02:06.677412 17573 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.677493 17573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.678814 17573 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:06.679474 17589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.679670 17573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:06.679776 17573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "2e1fa46a16d94d5db674ad47ed826c6b"
format_stamp: "Formatted at 2026-09-27 07:02:05 on dist-test-slave-mk3h"
server_key: "c5c19261734aa05defe0f0b4c7aeffc3"
server_key_iv: "a8032d3ed44132be1ab4d8d5109e9d22"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.680161 17573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.699518 17573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.701359 17573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.701521 17573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.705478 17573 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35101
I20260927 07:02:06.705508 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35101 every 8 connection(s)
I20260927 07:02:06.705875 17573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:06.707401 17642 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:02:06.708066 17642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap starting.
I20260927 07:02:06.710595 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17573
I20260927 07:02:06.710873 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:40803
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=36965
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:06.716637 17642 log.cc:824] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:06.718611 17642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:06.718896 17642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap complete.
I20260927 07:02:06.721205 17642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:06.721423 17642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Initialized, Role: FOLLOWER
I20260927 07:02:06.721597 17642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:06.721654 17642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:06.721728 17642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:06.721774 17642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:02:06.722715 17642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:06.722797 17642 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1fa46a16d94d5db674ad47ed826c6b; no voters:
I20260927 07:02:06.723009 17642 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:02:06.723042 17646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:02:06.723156 17646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 LEADER]: Becoming Leader. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Running, Role: LEADER
I20260927 07:02:06.723266 17642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:06.723243 17646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:06.723587 17647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:06.723584 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1fa46a16d94d5db674ad47ed826c6b. Latest consensus state: current_term: 2 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:06.723781 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:06.723922 17647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:06.724401 17653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:06.725147 17653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:02:06.725243 17653 catalog_manager.cc:1259] Loaded cluster ID: f3fa951941ec4d6a8109635cbd56a29e
I20260927 07:02:06.725282 17653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:06.725517 17653 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 07:02:06.728369 17653 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 07:02:06.730026 17653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:06.730221 17653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Loaded TSK: 0
I20260927 07:02:06.730433 17653 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:02:06.803473 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.803759 17644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.803799 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.805039 17644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.805104 17644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.805148 17644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:02:06.806624 17644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:40803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=36965
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.806874 17644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.807114 17644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.811980 17670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.812014 17669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.812079 17644 server_base.cc:1034] running on GCE node
W20260927 07:02:06.812264 17672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.812500 17644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.812763 17644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.813943 17644 hybrid_clock.cc:648] HybridClock initialized: now 1790492526813925 us; error 31 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.816695 17644 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:02:06.817159 17644 webserver.cc:460] Webserver started at http://127.3.57.1:36965/ using document root <none> and password file <none>
I20260927 07:02:06.817394 17644 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.817456 17644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.818897 17644 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:06.819610 17679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.819772 17644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:06.819849 17644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "a2687d43e81f430d98403242a40856ee"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "de2acc551b0a818cf68afd92dc4a4391"
server_key_iv: "a98c0814fa883df07c3c3755495c1ed9"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.820173 17644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.829931 17644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.832108 17644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.832255 17644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.832481 17644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:06.832917 17686 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:02:06.834700 17644 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:02:06.834743 17644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:02:06.834785 17644 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:02:06.835346 17644 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:02:06.835403 17644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:06.835492 17686 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Bootstrap starting.
I20260927 07:02:06.842478 17644 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:40803
I20260927 07:02:06.842811 17644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:02:06.843752 17793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:40803 every 8 connection(s)
I20260927 07:02:06.846052 17686 log.cc:824] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:06.846278 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17644
I20260927 07:02:06.846542 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:40677
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=33755
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:06.849191 17686 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:06.849459 17686 tablet_bootstrap.cc:492] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Bootstrap complete.
I20260927 07:02:06.850440 17686 ts_tablet_manager.cc:1397] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260927 07:02:06.851393 17686 raft_consensus.cc:348] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.851672 17686 raft_consensus.cc:729] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2687d43e81f430d98403242a40856ee, State: Initialized, Role: FOLLOWER
I20260927 07:02:06.851801 17686 consensus_queue.cc:260] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } }
I20260927 07:02:06.852074 17686 ts_tablet_manager.cc:1428] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:06.860455 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.844400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:53297 (local address 127.3.57.62:35101)
0927 07:02:06.847637 (+ 3237us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.847646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.847682 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:02:06.848401 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.848407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.848475 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.848605 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.852633 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.853471 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.854437 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.854853 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.855003 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.857075 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.857079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.857079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.857092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.858674 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.859129 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.859132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.859132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.859227 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.859482 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.859483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.859483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.859745 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.859850 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.860128 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.860256 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3191,"thread_start_us":83,"threads_started":1}
I20260927 07:02:06.860464 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.844332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:53297)
0927 07:02:06.846321 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.846363 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.848146 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.851800 (+ 3654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.851804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.852266 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:06.852487 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.852492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.853593 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.853594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.854338 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.854340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.854976 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.854977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.855108 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.855667 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:06.855675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:06.856947 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:06.858794 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.858801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.858802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.859024 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.859293 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.859294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.859295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.859405 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.859814 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:06.859820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:06.859969 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.860065 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.860239 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"thread_start_us":90,"threads_started":1}
I20260927 07:02:06.861474 17794 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:06.861560 17794 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:06.861719 17794 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:06.862408 17606 ts_manager.cc:194] Registered new tserver with Master: a2687d43e81f430d98403242a40856ee (127.3.57.1:40803)
I20260927 07:02:06.863489 17606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:53297
I20260927 07:02:06.864315 17794 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
I20260927 07:02:06.870796 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.862952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40803 (local address 127.0.0.1:38652)
0927 07:02:06.867654 (+ 4702us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.867694 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.867776 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.867974 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.867977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.867979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:06.868068 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.868070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.868886 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.868887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.869770 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.869773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.870497 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.870498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.870591 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.870604 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.870678 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.870729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4674,"thread_start_us":98,"threads_started":1}
I20260927 07:02:06.870795 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.863196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.3.57.1:40803)
0927 07:02:06.863353 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.863358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.867758 (+ 4400us) server_negotiation.cc:407] Connection header received
0927 07:02:06.867852 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.867854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.867868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.867958 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:06.868154 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.868794 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.869861 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.870313 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.870495 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.870605 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.870691 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.870728 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260927 07:02:06.871414 17728 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e27a473828d417f8a916ea636fdc57c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:06 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:38652
I20260927 07:02:06.871974 17803 tablet_replica.cc:330] stopping tablet replica
I20260927 07:02:06.872087 17803 raft_consensus.cc:2229] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:06.872152 17803 raft_consensus.cc:2258] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:06.872448 17803 ts_tablet_manager.cc:1905] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:06.874297 17803 ts_tablet_manager.cc:1918] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:06.874392 17803 log.cc:1196] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/6e27a473828d417f8a916ea636fdc57c
I20260927 07:02:06.874617 17803 ts_tablet_manager.cc:1939] T 6e27a473828d417f8a916ea636fdc57c P a2687d43e81f430d98403242a40856ee: Deleting consensus metadata
I20260927 07:02:06.875066 17592 catalog_manager.cc:4841] TS a2687d43e81f430d98403242a40856ee (127.3.57.1:40803): tablet 6e27a473828d417f8a916ea636fdc57c (table test-table [id=01c3af1e8ebb4023aa6de354ef72fa8c]) successfully deleted
W20260927 07:02:06.935729 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:06.935906 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:06.935927 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:06.937014 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:06.937053 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:06.937079 17799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:02:06.938508 17799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:40677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=33755
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:06.938719 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:06.938944 17799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:06.943786 17808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.943794 17811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:06.943943 17809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:06.944043 17799 server_base.cc:1034] running on GCE node
I20260927 07:02:06.944211 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:06.944386 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:06.945573 17799 hybrid_clock.cc:648] HybridClock initialized: now 1790492526945562 us; error 28 us; skew 500 ppm
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:06.948002 17799 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:02:06.948339 17799 webserver.cc:460] Webserver started at http://127.3.57.2:33755/ using document root <none> and password file <none>
I20260927 07:02:06.948508 17799 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:06.948554 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:06.949726 17799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:06.950259 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.950429 17799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.950532 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "45931ab91b2f41d9af4768311ee9263c"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "589a364392832649d7f7b96d0c61dc34"
server_key_iv: "e52b3a8d9eb144ac431e7034685a21e1"
server_key_version: "encryptionkey@0"
I20260927 07:02:06.950830 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:06.960032 17799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:06.962172 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:06.962325 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:06.962543 17799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:06.962844 17799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:06.962895 17799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.962925 17799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:06.962952 17799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:06.968345 17799 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:40677
I20260927 07:02:06.968415 17931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:40677 every 8 connection(s)
I20260927 07:02:06.968734 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:02:06.971768 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17799
I20260927 07:02:06.972064 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:40277
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=38473
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:02:06 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492526, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:06.978375 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.969422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:38247 (local address 127.3.57.62:35101)
0927 07:02:06.969489 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:06.969491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:06.969688 (+ 197us) server_negotiation.cc:407] Connection header received
0927 07:02:06.970139 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:06.970140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:06.970152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:06.970201 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:06.971095 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.971606 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.973056 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.973249 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.973326 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.975611 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:06.975613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:06.975613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:06.975620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:06.976996 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.977492 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.977492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.977493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.977539 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:06.977785 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:06.977786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:06.977786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:06.977878 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:06.977924 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:06.978246 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 07:02:06.978314 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:02:06.978746 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:06.969420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:38247)
0927 07:02:06.969566 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:06.969599 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:06.970059 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:06.970261 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:06.970264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:06.970834 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:06.971021 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.971026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.971690 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.971691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.972975 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:06.972978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:06.973349 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:06.973350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:06.973463 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:06.973903 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:06.973911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:06.975485 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:06.977081 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.977087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.977090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.977408 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.977584 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:06.977585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:06.977585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:06.977721 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:06.977941 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:06.977948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:06.978111 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 07:02:06.978236 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:02:06.978413 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260927 07:02:06.979228 17932 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:06.979322 17932 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:06.979521 17932 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:06.979894 17606 ts_manager.cc:194] Registered new tserver with Master: 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677)
I20260927 07:02:06.980248 17606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:38247
W20260927 07:02:07.065187 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:07.065413 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:07.065446 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:07.066689 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:07.066737 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:07.066766 17935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:02:07.068487 17935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:40277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=38473
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:07.068710 17935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:07.068956 17935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:07.074870 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.074896 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.074908 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.075140 17935 server_base.cc:1034] running on GCE node
I20260927 07:02:07.075340 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:07.075652 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:07.077072 17935 hybrid_clock.cc:648] HybridClock initialized: now 1790492527077045 us; error 40 us; skew 500 ppm
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:07.080291 17935 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:02:07.081012 17935 webserver.cc:460] Webserver started at http://127.3.57.3:38473/ using document root <none> and password file <none>
I20260927 07:02:07.081233 17935 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:07.081295 17935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:07.082989 17935 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:02:07.084151 17950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.084419 17935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.085400 17935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "8269c02d3360426d80d917330316aea8"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "ce25383ec5a749518bc2bfbc0e266456"
server_key_iv: "893a5808efc5831980e40c27ebf5775a"
server_key_version: "encryptionkey@0"
I20260927 07:02:07.085817 17935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:07.114173 17935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:07.116724 17935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:07.116838 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:07.117045 17935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:07.117345 17935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:07.117383 17935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.117408 17935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:07.117419 17935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.127264 17935 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:40277
I20260927 07:02:07.127308 18063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:40277 every 8 connection(s)
I20260927 07:02:07.127723 17935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:02:07.128481 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 17935
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:07.139357 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.129749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.3.57.62:35101)
0927 07:02:07.129873 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.129875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.129934 (+ 59us) server_negotiation.cc:407] Connection header received
0927 07:02:07.130005 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.130006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.130016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.130081 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:07.130787 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.131436 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.132403 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.132727 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.132805 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.134475 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:07.134476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:07.134476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:07.134483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:07.137284 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:07.138141 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:07.138142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:07.138143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:07.138211 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:07.138496 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:07.138497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:07.138498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:07.138614 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:07.139109 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.139153 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.139241 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 07:02:07.142274 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.128972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:54701 (local address 127.3.57.62:35101)
0927 07:02:07.129026 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.129029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.129322 (+ 293us) server_negotiation.cc:407] Connection header received
0927 07:02:07.129971 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.129973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.129986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.130043 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:07.131174 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.131787 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.133158 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.133343 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.133426 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.136046 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:07.136048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:07.136048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:07.136054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:07.138908 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:07.140503 (+ 1595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:07.140504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:07.140504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:07.140572 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:07.141439 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:07.141440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:07.141441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:07.141582 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:07.141670 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.142088 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.142179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:02:07.142540 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.128989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:54701)
0927 07:02:07.129171 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.129214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.129828 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.130119 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:07.130123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:07.130764 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:07.131033 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.131039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.131905 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.131906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.133069 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.133073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.133438 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.133439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.133564 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.134092 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:07.134101 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:07.135919 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:07.139010 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:07.139018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:07.139021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:07.139329 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:07.140954 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:07.140955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:07.140955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:07.141137 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:07.141663 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:07.141670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:07.141876 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 07:02:07.142042 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 07:02:07.142229 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260927 07:02:07.143112 18064 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35101
I20260927 07:02:07.143239 18064 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:07.143436 18064 heartbeater.cc:511] Master 127.3.57.62:35101 requested a full tablet report, sending...
I20260927 07:02:07.143962 17606 ts_manager.cc:194] Registered new tserver with Master: 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
I20260927 07:02:07.144346 17606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:54701
I20260927 07:02:07.154680 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:02:07.158581 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.155379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.3.57.62:35101)
0927 07:02:07.155452 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.155454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.155727 (+ 273us) server_negotiation.cc:407] Connection header received
0927 07:02:07.155778 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.155779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.155819 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.155883 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:07.156101 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.156604 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.157759 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.157958 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.158014 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.158220 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:07.158333 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:02:07.158400 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.158505 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.158523 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:02:07.158875 17606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:02:07.160607 17606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:02:07.163501 17728 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 (DEFAULT_TABLE table=test-table [id=230caecfc2dc4b93ab6afb47e5624838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:07.163743 17728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba90cf623e44e4c9530e739f2e1cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:07.165433 17686 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Bootstrap starting.
I20260927 07:02:07.165738 17686 tablet_bootstrap.cc:654] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:07.166306 17686 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: No bootstrap required, opened a new log
I20260927 07:02:07.166356 17686 ts_tablet_manager.cc:1397] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.166467 17686 raft_consensus.cc:348] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.166565 17686 raft_consensus.cc:374] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:07.166620 17686 raft_consensus.cc:729] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2687d43e81f430d98403242a40856ee, State: Initialized, Role: FOLLOWER
I20260927 07:02:07.166674 17686 consensus_queue.cc:260] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.166769 17686 ts_tablet_manager.cc:1428] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.167485 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.162867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:40677 (local address 127.0.0.1:58496)
0927 07:02:07.162929 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.162940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.162981 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.163615 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:07.163616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:07.163618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:07.163657 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.163659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.164989 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.164990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.166189 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.166192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.167177 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.167178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.167242 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.167245 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:02:07.167277 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 07:02:07.167335 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":57}
I20260927 07:02:07.167590 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.163097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.3.57.3:40277)
0927 07:02:07.163295 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.163300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.163317 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:02:07.163363 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.163365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.163377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.163472 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:07.163643 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.164709 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.166400 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.167067 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.167139 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.167200 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.167443 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.167486 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260927 07:02:07.167675 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.162996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.3.57.2:40677)
0927 07:02:07.163408 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.163412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.163423 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:02:07.163463 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.163465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.163480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.163740 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:07.163827 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.164866 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.166293 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.167067 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.167422 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.167488 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.167513 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.167588 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":147,"threads_started":1}
I20260927 07:02:07.167702 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.162990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40277 (local address 127.0.0.1:34474)
0927 07:02:07.163197 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.163208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.163259 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.163486 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:07.163487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:07.163490 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:07.163539 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.163541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.165016 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.165017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.166222 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.166224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.167228 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.167228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.167301 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.167304 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:02:07.167347 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 07:02:07.167377 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":11,"thread_start_us":132,"threads_started":1}
I20260927 07:02:07.168283 17866 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 (DEFAULT_TABLE table=test-table [id=230caecfc2dc4b93ab6afb47e5624838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:07.168601 17866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba90cf623e44e4c9530e739f2e1cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:07.171090 18074 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Bootstrap starting.
I20260927 07:02:07.171422 17998 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 (DEFAULT_TABLE table=test-table [id=230caecfc2dc4b93ab6afb47e5624838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:07.171723 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba90cf623e44e4c9530e739f2e1cb90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:07.171726 18074 tablet_bootstrap.cc:654] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:07.172274 18074 log.cc:824] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:07.172976 18074 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: No bootstrap required, opened a new log
I20260927 07:02:07.173027 18074 ts_tablet_manager.cc:1397] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:02:07.174383 18076 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Bootstrap starting.
I20260927 07:02:07.174324 18074 raft_consensus.cc:348] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.174512 18074 raft_consensus.cc:374] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:07.174546 18074 raft_consensus.cc:729] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45931ab91b2f41d9af4768311ee9263c, State: Initialized, Role: FOLLOWER
I20260927 07:02:07.174665 18074 consensus_queue.cc:260] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.174942 18074 ts_tablet_manager.cc:1428] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:02:07.174961 17932 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
I20260927 07:02:07.175028 18076 tablet_bootstrap.cc:654] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:07.175661 18076 log.cc:824] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:07.176373 18076 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: No bootstrap required, opened a new log
I20260927 07:02:07.176422 18076 ts_tablet_manager.cc:1397] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:02:07.177641 18076 raft_consensus.cc:348] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.177767 18076 raft_consensus.cc:374] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:07.177788 18076 raft_consensus.cc:729] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8269c02d3360426d80d917330316aea8, State: Initialized, Role: FOLLOWER
I20260927 07:02:07.177862 18076 consensus_queue.cc:260] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.178035 18076 ts_tablet_manager.cc:1428] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:02:07.178089 18064 heartbeater.cc:503] Master 127.3.57.62:35101 was elected leader, sending a full tablet report...
W20260927 07:02:07.219280 17933 tablet.cc:2369] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:02:07.289220 18077 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:07.289359 18077 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.289669 18077 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277), a2687d43e81f430d98403242a40856ee (127.3.57.1:40803)
I20260927 07:02:07.293503 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.289912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:58261 (local address 127.3.57.3:40277)
0927 07:02:07.289962 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.289964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.289995 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:02:07.290024 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.290025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.290037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.290091 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:07.290259 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.291071 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.292588 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.293142 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.293225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.293235 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.293398 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.293435 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:02:07.293503 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.290019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:54527 (local address 127.3.57.1:40803)
0927 07:02:07.290083 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.290084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.290168 (+ 84us) server_negotiation.cc:407] Connection header received
0927 07:02:07.290243 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.290244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.290255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.290344 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:07.290480 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.291146 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.292589 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.293142 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.293225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.293235 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.293398 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.293435 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:02:07.293516 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.289820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40277 (local address 127.3.57.2:58261)
0927 07:02:07.289880 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.289892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.289942 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.290104 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:07.290105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:07.290108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:07.290178 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.290179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.291146 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.291147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.292497 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.292499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.293236 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.293237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.293314 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.293316 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:02:07.293372 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:02:07.293416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":58}
I20260927 07:02:07.293524 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.289947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40803 (local address 127.3.57.2:54527)
0927 07:02:07.290097 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.290106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.290157 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.290358 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:07.290358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:07.290360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:07.290412 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.290414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.291216 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.291216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.292490 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:07.292494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.293237 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.293237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:07.293311 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.293316 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:02:07.293372 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 07:02:07.293416 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":6,"thread_start_us":97,"threads_started":1}
I20260927 07:02:07.293875 18018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba90cf623e44e4c9530e739f2e1cb90" candidate_uuid: "45931ab91b2f41d9af4768311ee9263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8269c02d3360426d80d917330316aea8" is_pre_election: true
I20260927 07:02:07.293987 18018 raft_consensus.cc:2454] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45931ab91b2f41d9af4768311ee9263c in term 0.
I20260927 07:02:07.294196 17821 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8; no voters:
I20260927 07:02:07.294318 18077 raft_consensus.cc:2781] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:02:07.294373 18077 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:02:07.294389 18077 raft_consensus.cc:3037] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:07.294514 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba90cf623e44e4c9530e739f2e1cb90" candidate_uuid: "45931ab91b2f41d9af4768311ee9263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2687d43e81f430d98403242a40856ee" is_pre_election: true
I20260927 07:02:07.294663 17748 raft_consensus.cc:2454] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45931ab91b2f41d9af4768311ee9263c in term 0.
I20260927 07:02:07.295099 18077 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.295215 18077 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [CANDIDATE]: Term 1 election: Requested vote from peers 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277), a2687d43e81f430d98403242a40856ee (127.3.57.1:40803)
I20260927 07:02:07.295421 18018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba90cf623e44e4c9530e739f2e1cb90" candidate_uuid: "45931ab91b2f41d9af4768311ee9263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8269c02d3360426d80d917330316aea8"
I20260927 07:02:07.295414 17748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba90cf623e44e4c9530e739f2e1cb90" candidate_uuid: "45931ab91b2f41d9af4768311ee9263c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2687d43e81f430d98403242a40856ee"
I20260927 07:02:07.295485 18018 raft_consensus.cc:3037] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:07.295502 17748 raft_consensus.cc:3037] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:07.296151 18018 raft_consensus.cc:2454] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45931ab91b2f41d9af4768311ee9263c in term 1.
I20260927 07:02:07.296243 17748 raft_consensus.cc:2454] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45931ab91b2f41d9af4768311ee9263c in term 1.
I20260927 07:02:07.296314 17821 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8; no voters:
I20260927 07:02:07.296428 18077 raft_consensus.cc:2781] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:07.296540 18077 raft_consensus.cc:686] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 1 LEADER]: Becoming Leader. State: Replica: 45931ab91b2f41d9af4768311ee9263c, State: Running, Role: LEADER
I20260927 07:02:07.296636 18077 consensus_queue.cc:237] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.297390 17605 catalog_manager.cc:5495] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45931ab91b2f41d9af4768311ee9263c (127.3.57.2). New cstate: current_term: 1 leader_uuid: "45931ab91b2f41d9af4768311ee9263c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260927 07:02:07.311690 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.309642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.3.57.2:40677)
0927 07:02:07.309684 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:07.309685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:07.309693 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:07.309769 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:07.309770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:07.309785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:07.309832 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:07.309902 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.310339 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.311003 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:07.311173 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:07.311220 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:07.311339 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:07.311433 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:02:07.311491 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:07.311544 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:02:07.311583 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:02:07.313668 18018 raft_consensus.cc:1261] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Refusing update from remote peer 45931ab91b2f41d9af4768311ee9263c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:07.313668 17748 raft_consensus.cc:1261] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Refusing update from remote peer 45931ab91b2f41d9af4768311ee9263c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:07.313961 18077 consensus_queue.cc:1035] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:07.314107 18081 consensus_queue.cc:1035] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:07.316331 18084 mvcc.cc:205] Tried to move back new op lower bound from 7333857391874207744 to 7333857391807414272. Current Snapshot: MvccSnapshot[applied={T|T < 7333857391874207744}]
I20260927 07:02:07.316459 18086 mvcc.cc:205] Tried to move back new op lower bound from 7333857391874207744 to 7333857391807414272. Current Snapshot: MvccSnapshot[applied={T|T < 7333857391874207744}]
I20260927 07:02:07.317165 18085 mvcc.cc:205] Tried to move back new op lower bound from 7333857391874207744 to 7333857391807414272. Current Snapshot: MvccSnapshot[applied={T|T < 7333857391874207744}]
I20260927 07:02:07.318351 17606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:07.318437 17606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:07.319073 17606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Sending DeleteTablet for 3 replicas of tablet 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:07.319352 17998 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:34474
I20260927 07:02:07.319350 17866 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260927 07:02:07.319531 17728 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba90cf623e44e4c9530e739f2e1cb90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:07 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:38652
I20260927 07:02:07.319577 18095 tablet_replica.cc:330] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: stopping tablet replica
I20260927 07:02:07.319635 17803 tablet_replica.cc:330] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: stopping tablet replica
I20260927 07:02:07.319692 17803 raft_consensus.cc:2229] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:07.319720 18095 raft_consensus.cc:2229] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:02:07.319754 17803 raft_consensus.cc:2258] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:07.319763 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17644
I20260927 07:02:07.319854 18095 raft_consensus.cc:2258] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:07.320057 18094 tablet_replica.cc:330] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: stopping tablet replica
I20260927 07:02:07.320158 18094 raft_consensus.cc:2229] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:07.320299 18094 raft_consensus.cc:2258] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:07.320749 18094 ts_tablet_manager.cc:1905] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:07.321403 18095 ts_tablet_manager.cc:1905] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:07.322407 18094 ts_tablet_manager.cc:1918] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:07.322507 18094 log.cc:1196] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:07.322944 18094 ts_tablet_manager.cc:1939] T 2ba90cf623e44e4c9530e739f2e1cb90 P 8269c02d3360426d80d917330316aea8: Deleting consensus metadata
I20260927 07:02:07.323504 17592 catalog_manager.cc:4841] TS 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): tablet 2ba90cf623e44e4c9530e739f2e1cb90 (table test-table [id=230caecfc2dc4b93ab6afb47e5624838]) successfully deleted
I20260927 07:02:07.323668 18095 ts_tablet_manager.cc:1918] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:07.323741 18095 log.cc:1196] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:07.325388 18095 ts_tablet_manager.cc:1939] T 2ba90cf623e44e4c9530e739f2e1cb90 P 45931ab91b2f41d9af4768311ee9263c: Deleting consensus metadata
I20260927 07:02:07.325745 17590 catalog_manager.cc:4841] TS 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): tablet 2ba90cf623e44e4c9530e739f2e1cb90 (table test-table [id=230caecfc2dc4b93ab6afb47e5624838]) successfully deleted
W20260927 07:02:07.327484 17592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:40803 (error 108)
I20260927 07:02:07.327876 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17799
I20260927 07:02:07.327943 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.327831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:40803 (local address 127.0.0.1:38654)
0927 07:02:07.327871 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.327908 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 07:02:07.328459 17592 catalog_manager.cc:4584] TS a2687d43e81f430d98403242a40856ee (127.3.57.1:40803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ba90cf623e44e4c9530e739f2e1cb90: Network error: Client connection negotiation failed: client connection to 127.3.57.1:40803: connect: Connection refused (error 111)
I20260927 07:02:07.333003 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17935
I20260927 07:02:07.337240 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 17573
I20260927 07:02:07.342484 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--webserver_interface=127.3.57.62
--webserver_port=44739
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:07.419602 18097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:07.419785 18097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:07.419806 18097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:07.420928 18097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:07.420967 18097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:07.420981 18097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:07.421018 18097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:07.421046 18097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:07.422447 18097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=44739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:07.422676 18097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:07.422894 18097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:07.427574 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.427640 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.427736 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.427597 18097 server_base.cc:1034] running on GCE node
I20260927 07:02:07.427980 18097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:07.428310 18097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:07.429482 18097 hybrid_clock.cc:648] HybridClock initialized: now 1790492527429477 us; error 37 us; skew 500 ppm
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:07.432044 18097 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:07.432461 18097 webserver.cc:460] Webserver started at http://127.3.57.62:44739/ using document root <none> and password file <none>
I20260927 07:02:07.432696 18097 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:07.432765 18097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:07.434072 18097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.434701 18112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.434880 18097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:07.434981 18097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "2e1fa46a16d94d5db674ad47ed826c6b"
format_stamp: "Formatted at 2026-09-27 07:02:05 on dist-test-slave-mk3h"
server_key: "c5c19261734aa05defe0f0b4c7aeffc3"
server_key_iv: "a8032d3ed44132be1ab4d8d5109e9d22"
server_key_version: "encryptionkey@0"
I20260927 07:02:07.435307 18097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:07.451903 18097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:07.453630 18097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:07.453791 18097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:07.458017 18097 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35101
I20260927 07:02:07.458048 18164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35101 every 8 connection(s)
I20260927 07:02:07.458412 18097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:07.459874 18165 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:02:07.460536 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap starting.
I20260927 07:02:07.466984 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18097
I20260927 07:02:07.467255 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:40803
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=36965
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:07.468540 18165 log.cc:824] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:07.471405 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:07.471753 18165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b: Bootstrap complete.
I20260927 07:02:07.475621 18165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:07.476006 18165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Initialized, Role: FOLLOWER
I20260927 07:02:07.476177 18165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:07.476272 18165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:07.476331 18165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:07.476389 18165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:02:07.477876 18165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:07.477995 18165 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1fa46a16d94d5db674ad47ed826c6b; no voters:
I20260927 07:02:07.478158 18165 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:02:07.478225 18169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:02:07.478402 18169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [term 3 LEADER]: Becoming Leader. State: Replica: 2e1fa46a16d94d5db674ad47ed826c6b, State: Running, Role: LEADER
I20260927 07:02:07.478442 18165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:07.478543 18169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } }
I20260927 07:02:07.478806 18170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:07.478804 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1fa46a16d94d5db674ad47ed826c6b. Latest consensus state: current_term: 3 leader_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1fa46a16d94d5db674ad47ed826c6b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35101 } } }
I20260927 07:02:07.478945 18170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:07.478979 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1fa46a16d94d5db674ad47ed826c6b [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:07.479207 18178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:07.480159 18178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:02:07.480283 18178 catalog_manager.cc:1259] Loaded cluster ID: f3fa951941ec4d6a8109635cbd56a29e
I20260927 07:02:07.480338 18178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 07:02:07.480557 18178 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd6b14069ad google::LogMessage::Fail() at ??:0
@ 0x7fd6b140988a google::LogMessage::SendToLog() at ??:0
@ 0x7fd6b14064de google::LogMessage::Flush() at ??:0
@ 0x7fd6b1407a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd6b93270f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd6b93308b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd6b32d40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd6b32c9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd6b412d6db start_thread at ??:0
@ 0x7fd6aff0461f clone at ??:0
W20260927 07:02:07.559964 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:07.560173 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:07.560209 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:07.561313 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:07.561378 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:07.561422 18167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:02:07.562939 18167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:40803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=36965
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:07.563184 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:07.563403 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:07.568189 18202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.568326 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.568231 18201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.568427 18167 server_base.cc:1034] running on GCE node
I20260927 07:02:07.568600 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:07.568854 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:07.570015 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790492527570004 us; error 30 us; skew 500 ppm
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:07.572575 18167 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:02:07.573030 18167 webserver.cc:460] Webserver started at http://127.3.57.1:36965/ using document root <none> and password file <none>
I20260927 07:02:07.573246 18167 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:07.573312 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:07.574618 18167 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.575309 18211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.575481 18167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.575575 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "a2687d43e81f430d98403242a40856ee"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "de2acc551b0a818cf68afd92dc4a4391"
server_key_iv: "a98c0814fa883df07c3c3755495c1ed9"
server_key_version: "encryptionkey@0"
I20260927 07:02:07.575904 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:07.609421 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:07.611760 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:07.611931 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:07.612167 18167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:07.612609 18218 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:02:07.614490 18167 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:02:07.614549 18167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:02:07.614596 18167 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:02:07.615212 18167 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:02:07.615266 18167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:07.615382 18218 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Bootstrap starting.
I20260927 07:02:07.622363 18167 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:40803
I20260927 07:02:07.622463 18325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:40803 every 8 connection(s)
I20260927 07:02:07.622767 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:02:07.622987 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18167
I20260927 07:02:07.623260 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:40677
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=33755
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:07.626484 18218 log.cc:824] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:07.628582 18218 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:07.628892 18218 tablet_bootstrap.cc:492] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Bootstrap complete.
I20260927 07:02:07.629827 18218 ts_tablet_manager.cc:1397] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260927 07:02:07.631392 18218 raft_consensus.cc:348] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.631680 18218 raft_consensus.cc:729] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2687d43e81f430d98403242a40856ee, State: Initialized, Role: FOLLOWER
I20260927 07:02:07.631942 18218 consensus_queue.cc:260] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:07.632270 18218 ts_tablet_manager.cc:1428] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260927 07:02:07.690850 18329 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:02:07.625301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:57919)
0927 07:02:07.630002 (+ 4701us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.630046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:07.631036 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:07.690645 (+ 59609us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4636,"thread_start_us":88,"threads_started":1}
W20260927 07:02:07.691098 18327 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:35101 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:35101: Client connection negotiation failed: client connection to 127.3.57.62:35101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:02:07.691462 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.691352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:57595)
0927 07:02:07.691406 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.691444 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:02:07.691780 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.691699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:39315)
0927 07:02:07.691744 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.691766 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:02:07.691923 18327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:02:07.723378 18326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:07.723619 18326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:07.723662 18326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:07.724797 18326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:07.724861 18326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:07.724905 18326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:02:07.726385 18326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:40677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=33755
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:07.726639 18326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:07.726854 18326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:07.731751 18337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.731784 18336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.731938 18326 server_base.cc:1034] running on GCE node
W20260927 07:02:07.731748 18339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.732249 18326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:07.732548 18326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:07.733731 18326 hybrid_clock.cc:648] HybridClock initialized: now 1790492527733718 us; error 24 us; skew 500 ppm
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:07.737418 18326 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:02:07.737964 18326 webserver.cc:460] Webserver started at http://127.3.57.2:33755/ using document root <none> and password file <none>
I20260927 07:02:07.738273 18326 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:07.738350 18326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:07.740150 18326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:02:07.740926 18346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.741115 18326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.741196 18326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "45931ab91b2f41d9af4768311ee9263c"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "589a364392832649d7f7b96d0c61dc34"
server_key_iv: "e52b3a8d9eb144ac431e7034685a21e1"
server_key_version: "encryptionkey@0"
I20260927 07:02:07.741626 18326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:07.780138 18326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:07.782135 18326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:07.782305 18326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:07.782552 18326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:07.782866 18326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:07.782924 18326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.782969 18326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:07.783000 18326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.789153 18326 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:40677
I20260927 07:02:07.789207 18459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:40677 every 8 connection(s)
I20260927 07:02:07.789554 18326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:02:07.790592 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.790163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:33841)
0927 07:02:07.790385 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.790455 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":134,"threads_started":1}
W20260927 07:02:07.791007 18460 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:35101 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:35101: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
I20260927 07:02:07.791273 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.791178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:49471)
0927 07:02:07.791227 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.791253 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 07:02:07.791599 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.791471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:48693)
0927 07:02:07.791511 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.791534 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 07:02:07.791741 18460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:02:07.799679 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18326
I20260927 07:02:07.799949 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:40277
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:35101
--webserver_port=38473
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:07.878854 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:07.879033 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:07.879055 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:07.880234 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:07.880283 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:07.880309 18463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:02:07.881712 18463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:40277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=38473
--tserver_master_addrs=127.3.57.62:35101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:07.881955 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:07.882167 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:07.887037 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:07.887087 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.887084 18463 server_base.cc:1034] running on GCE node
W20260927 07:02:07.887043 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:07.887513 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:07.887776 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:07.888918 18463 hybrid_clock.cc:648] HybridClock initialized: now 1790492527888905 us; error 28 us; skew 500 ppm
Sep 27 07:02:07 dist-test-slave-mk3h krb5kdc[17029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:07.891305 18463 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:02:07.891812 18463 webserver.cc:460] Webserver started at http://127.3.57.3:38473/ using document root <none> and password file <none>
I20260927 07:02:07.892030 18463 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:07.892074 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:07.893321 18463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:07.893903 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.894217 18463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:07.894284 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "8269c02d3360426d80d917330316aea8"
format_stamp: "Formatted at 2026-09-27 07:02:06 on dist-test-slave-mk3h"
server_key: "ce25383ec5a749518bc2bfbc0e266456"
server_key_iv: "893a5808efc5831980e40c27ebf5775a"
server_key_version: "encryptionkey@0"
I20260927 07:02:07.894586 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:07.920984 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:07.922706 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:07.922876 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:07.923085 18463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:07.923379 18463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:07.923445 18463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.923477 18463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:07.923504 18463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:07.929589 18463 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:40277
I20260927 07:02:07.930058 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:02:07.930143 18591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:40277 every 8 connection(s)
I20260927 07:02:07.931052 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.930733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:42891)
0927 07:02:07.930866 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.930930 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260927 07:02:07.931435 18592 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:35101 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:35101: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
I20260927 07:02:07.931738 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.931655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:51427)
0927 07:02:07.931698 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.931722 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 07:02:07.931990 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:07.931944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:55879)
0927 07:02:07.931963 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:07.931980 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 07:02:07.932106 18592 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:02:07.934711 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18463
I20260927 07:02:08.692440 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:08.692233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:42581)
0927 07:02:08.692349 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:08.692382 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260927 07:02:08.792222 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:08.792030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:53773)
0927 07:02:08.792145 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:08.792176 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260927 07:02:08.932610 18598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:08.932415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:57405)
0927 07:02:08.932535 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:08.932566 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260927 07:02:09.108042 18599 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:09.108170 18599 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:09.108445 18599 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
Sep 27 07:02:09 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.2@KRBTEST.COM
Sep 27 07:02:09 dist-test-slave-mk3h krb5kdc[17029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492527, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.3@KRBTEST.COM
I20260927 07:02:09.120960 18600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.108625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:40677 (local address 127.3.57.1:48123)
0927 07:02:09.108742 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:09.108750 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:09.108788 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:09.109799 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:09.109802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:09.110330 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:09.110593 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:09.110600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.112284 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.112285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:09.113618 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:09.113620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.114438 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.114440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:09.114589 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:09.115252 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:09.115256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:09.117069 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:09.119420 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:09.119426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:09.119428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:09.119712 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:09.119968 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:09.119969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:09.119969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:09.120070 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:09.120440 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:09.120446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:09.120568 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 07:02:09.120669 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 07:02:09.120868 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":126,"thread_start_us":61,"threads_started":1}
I20260927 07:02:09.121031 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.108721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:46435 (local address 127.3.57.3:40277)
0927 07:02:09.108899 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:09.108922 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:09.108950 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:02:09.109513 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:09.109518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:09.109575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:09.109733 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:09.110734 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.112146 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.113710 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.114332 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.114448 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:09.117550 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:09.117554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:09.117554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:09.117567 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:09.119536 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:09.119914 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:09.119916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:09.119917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:09.120013 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:09.120215 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:09.120216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:09.120217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:09.120442 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:09.120526 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:09.120748 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:02:09.120911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260927 07:02:09.121074 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.108610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:40277 (local address 127.3.57.1:46435)
0927 07:02:09.108668 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:09.108679 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:09.108732 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:09.109799 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:09.109802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:09.110330 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:09.110593 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:09.110600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.112294 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.112294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:09.113618 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:09.113620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.114440 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.114441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:09.114593 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:09.115231 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:09.115242 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:09.117444 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:09.119655 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:09.119656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:09.119656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:09.119832 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:09.120088 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:09.120089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:09.120089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:09.120152 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:09.120521 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:09.120522 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:09.120568 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 07:02:09.120674 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 07:02:09.120947 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":202}
I20260927 07:02:09.120975 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.108740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:48123 (local address 127.3.57.2:40677)
0927 07:02:09.108899 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:09.108910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:09.108950 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:02:09.109524 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:09.109526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:09.109575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:09.109733 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:09.110734 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.112146 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.113710 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:09.114332 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:09.114448 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:09.117183 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:09.117188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:09.117188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:09.117201 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:09.119340 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:09.119796 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:09.119799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:09.119800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:09.119900 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:09.120137 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:09.120138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:09.120139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:09.120364 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:09.120448 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:09.120723 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:02:09.120857 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
W20260927 07:02:09.121835 18214 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
W20260927 07:02:09.122213 18212 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:09.122272 18212 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2687d43e81f430d98403242a40856ee; no voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8
I20260927 07:02:09.122380 18599 raft_consensus.cc:2726] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:02:09.693251 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.692988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:41063)
0927 07:02:09.693150 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:09.693193 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260927 07:02:09.792790 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.792608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:51409)
0927 07:02:09.792715 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:09.792748 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260927 07:02:09.933184 18605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:09.933009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:47297)
0927 07:02:09.933118 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:09.933148 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 07:02:10.693902 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:10.693702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:56963)
0927 07:02:10.693822 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:10.693855 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260927 07:02:10.747920 18607 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:10.748008 18607 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:10.748208 18607 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
W20260927 07:02:10.748564 18214 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
W20260927 07:02:10.748642 18212 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:10.748742 18212 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2687d43e81f430d98403242a40856ee; no voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8
I20260927 07:02:10.748802 18607 raft_consensus.cc:2726] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:02:10.793390 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:10.793206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:43119)
0927 07:02:10.793315 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:10.793351 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260927 07:02:10.933863 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:10.933664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:44753)
0927 07:02:10.933793 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:10.933828 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260927 07:02:11.694541 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:11.694328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:37567)
0927 07:02:11.694463 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:11.694497 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260927 07:02:11.793953 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:11.793778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:56227)
0927 07:02:11.793888 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:11.793920 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260927 07:02:11.934444 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:11.934275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:42681)
0927 07:02:11.934379 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:11.934410 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 07:02:12.695274 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:12.695044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:60635)
0927 07:02:12.695175 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:12.695210 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260927 07:02:12.794575 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:12.794396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:41419)
0927 07:02:12.794510 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:12.794541 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 07:02:12.935091 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:12.934849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:50369)
0927 07:02:12.935022 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:12.935056 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260927 07:02:13.696014 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:13.695774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:47601)
0927 07:02:13.695929 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:13.695968 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260927 07:02:13.795250 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:13.795008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:43983)
0927 07:02:13.795174 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:13.795212 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260927 07:02:13.879772 18618 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:13.879865 18618 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:13.880086 18618 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
W20260927 07:02:13.880468 18212 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
W20260927 07:02:13.880549 18214 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:13.880611 18214 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2687d43e81f430d98403242a40856ee; no voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8
I20260927 07:02:13.880705 18618 raft_consensus.cc:2726] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:02:13.935901 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:13.935646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:53307)
0927 07:02:13.935822 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:13.935858 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
I20260927 07:02:14.696689 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:14.696441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:47717)
0927 07:02:14.696560 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:14.696630 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 07:02:14.795917 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:14.795734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:52303)
0927 07:02:14.795847 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:14.795879 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260927 07:02:14.936483 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:14.936294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:59003)
0927 07:02:14.936416 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:14.936449 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260927 07:02:15.697322 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:15.697116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:42299)
0927 07:02:15.697242 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:15.697277 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260927 07:02:15.796584 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:15.796351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:44853)
0927 07:02:15.796508 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:15.796546 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260927 07:02:15.937189 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:15.937000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:46857)
0927 07:02:15.937115 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:15.937148 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260927 07:02:15.994882 18626 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:15.994976 18626 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:15.995143 18626 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
W20260927 07:02:15.995533 18212 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
W20260927 07:02:15.995649 18214 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:15.995718 18214 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2687d43e81f430d98403242a40856ee; no voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8
I20260927 07:02:15.995810 18626 raft_consensus.cc:2726] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:02:16.697996 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:16.697751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:53609)
0927 07:02:16.697917 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:16.697953 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260927 07:02:16.797341 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:16.797124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:46675)
0927 07:02:16.797254 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:16.797300 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":52,"threads_started":1}
I20260927 07:02:16.937831 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:16.937633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:41149)
0927 07:02:16.937760 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:16.937794 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260927 07:02:17.698684 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:17.698489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:37307)
0927 07:02:17.698611 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:17.698642 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260927 07:02:17.797952 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:17.797769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:48927)
0927 07:02:17.797881 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:17.797913 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 07:02:17.938539 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:17.938292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:59077)
0927 07:02:17.938448 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:17.938489 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260927 07:02:18.586400 18633 raft_consensus.cc:482] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:18.586486 18633 raft_consensus.cc:504] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45931ab91b2f41d9af4768311ee9263c" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 40677 } } peers { permanent_uuid: "8269c02d3360426d80d917330316aea8" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 40277 } } peers { permanent_uuid: "a2687d43e81f430d98403242a40856ee" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 40803 } }
I20260927 07:02:18.586642 18633 leader_election.cc:288] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677), 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277)
W20260927 07:02:18.587021 18212 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45931ab91b2f41d9af4768311ee9263c (127.3.57.2:40677): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
W20260927 07:02:18.587101 18214 leader_election.cc:341] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8269c02d3360426d80d917330316aea8 (127.3.57.3:40277): Not found: Tablet not found: 2ba90cf623e44e4c9530e739f2e1cb90
I20260927 07:02:18.587175 18214 leader_election.cc:302] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2687d43e81f430d98403242a40856ee; no voters: 45931ab91b2f41d9af4768311ee9263c, 8269c02d3360426d80d917330316aea8
I20260927 07:02:18.587265 18633 raft_consensus.cc:2726] T 2ba90cf623e44e4c9530e739f2e1cb90 P a2687d43e81f430d98403242a40856ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:02:18.699332 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:18.699099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:43359)
0927 07:02:18.699241 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:18.699284 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260927 07:02:18.798564 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:18.798374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:38275)
0927 07:02:18.798495 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:18.798527 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260927 07:02:18.939189 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:18.938978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:48599)
0927 07:02:18.939109 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:18.939144 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260927 07:02:19.700055 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:19.699853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:44117)
0927 07:02:19.699973 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:19.700006 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260927 07:02:19.799144 18638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:19.798932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:45465)
0927 07:02:19.799069 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:19.799105 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260927 07:02:19.939973 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:19.939773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:44407)
0927 07:02:19.939909 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:19.939938 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260927 07:02:20.700654 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:20.700469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:38767)
0927 07:02:20.700581 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:20.700613 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 07:02:20.799788 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:20.799594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:44785)
0927 07:02:20.799716 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:20.799750 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 07:02:20.940577 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:20.940385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:53633)
0927 07:02:20.940502 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:20.940536 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260927 07:02:21.701529 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:21.701304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:44927)
0927 07:02:21.701445 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:21.701481 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260927 07:02:21.800420 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:21.800214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:39361)
0927 07:02:21.800351 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:21.800385 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260927 07:02:21.941190 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:21.940994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:60393)
0927 07:02:21.941120 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:21.941148 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260927 07:02:22.702360 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:22.702177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.1:57657)
0927 07:02:22.702289 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:22.702317 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260927 07:02:22.801069 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:22.800896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.2:45323)
0927 07:02:22.801000 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:22.801031 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260927 07:02:22.940769 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18167
I20260927 07:02:22.942271 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:22.941841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35101 (local address 127.3.57.3:37971)
0927 07:02:22.942177 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:22.942222 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":50,"threads_started":1}
I20260927 07:02:22.945690 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18326
I20260927 07:02:22.950012 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18463
2026-09-27T07:02:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18298 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:02:23 dist-test-slave-mk3h krb5kdc[18651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:02:23 dist-test-slave-mk3h krb5kdc[18651](info): set up 2 sockets
Sep 27 07:02:23 dist-test-slave-mk3h krb5kdc[18651](info): commencing operation
krb5kdc: starting...
W20260927 07:02:23.963516 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:02:24.074584 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:24.152287 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:24.152506 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:24.152545 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:24.153661 18664 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:02:24.153729 18664 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:02:24.153751 18664 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:02:24.153770 18664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:24.153800 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:24.153832 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:24.153863 18664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:24.153892 18664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:24.155336 18664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:24.155610 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:24.155846 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:24.160305 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.160312 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.160315 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:24.160497 18664 server_base.cc:1034] running on GCE node
I20260927 07:02:24.160755 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:24.161024 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:24.162161 18664 hybrid_clock.cc:648] HybridClock initialized: now 1790492544162146 us; error 32 us; skew 500 ppm
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:24.164539 18664 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:24.164933 18664 webserver.cc:460] Webserver started at http://127.3.57.62:35359/ using document root <none> and password file <none>
I20260927 07:02:24.165165 18664 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:24.165208 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:24.165349 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:24.166321 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "1a8f5d35938a4d0b99891ed240de11c2"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "488b1171be9483738c8a5432ae612460"
server_key_iv: "56286f54a335088412d8e2e30eae9d18"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.166666 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "1a8f5d35938a4d0b99891ed240de11c2"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "488b1171be9483738c8a5432ae612460"
server_key_iv: "56286f54a335088412d8e2e30eae9d18"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.168002 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:02:24.168744 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.168910 18664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:02:24.169011 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "1a8f5d35938a4d0b99891ed240de11c2"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "488b1171be9483738c8a5432ae612460"
server_key_iv: "56286f54a335088412d8e2e30eae9d18"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.169085 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:24.178987 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:24.180650 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:24.180799 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:24.184592 18664 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37367
I20260927 07:02:24.184619 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37367 every 8 connection(s)
I20260927 07:02:24.184942 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:24.185683 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:24.187840 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap starting.
I20260927 07:02:24.188423 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18664
I20260927 07:02:24.188460 18732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:24.188527 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:02:24.188767 3300 external_mini_cluster.cc:1428] Setting key 62a13b5b94bea959a6a07e18844b0e4a
I20260927 07:02:24.188776 18732 log.cc:824] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:24.189502 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: No bootstrap required, opened a new log
I20260927 07:02:24.191468 18732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:24.191650 18732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:24.191679 18732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Initialized, Role: FOLLOWER
I20260927 07:02:24.191869 18732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:24.191964 18732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:24.192010 18732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:24.192058 18732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:24.192562 18732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:24.192675 18732 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8f5d35938a4d0b99891ed240de11c2; no voters:
I20260927 07:02:24.192869 18732 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:02:24.192894 18737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:24.193003 18737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Running, Role: LEADER
I20260927 07:02:24.193135 18737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:24.193138 18732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:24.193688 18738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:02:24.193774 18738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:24.193938 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8f5d35938a4d0b99891ed240de11c2. Latest consensus state: current_term: 1 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:02:24.194000 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:24.194265 18742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:24.194756 18742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:24.197633 18742 catalog_manager.cc:1347] Generated new cluster ID: cb5d9e9cd4024881b59e6c2689868e4a
I20260927 07:02:24.197681 18742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:24.199373 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.189260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.3.57.62:37367)
0927 07:02:24.189432 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.189442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.189475 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:02:24.189915 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.189921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.189964 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.190100 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:24.190725 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.191638 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.192478 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.193029 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.193407 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.195984 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:24.195988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:24.195989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:24.196000 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:24.197824 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.198209 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.198212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.198212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.198304 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.198512 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.198513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.198513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.198774 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:24.198911 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.199051 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.199174 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260927 07:02:24.223731 18742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:02:24.224501 18742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:24.230494 18742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Generated new TSK 0
I20260927 07:02:24.230680 18742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:02:24.257815 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:24.338438 18758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:24.338672 18758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:24.338714 18758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:24.339955 18758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:24.340018 18758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:24.340065 18758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:02:24.341552 18758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:24.341820 18758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:24.342052 18758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:24.347261 18763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.347262 18764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:24.347307 18758 server_base.cc:1034] running on GCE node
W20260927 07:02:24.347328 18766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:24.347834 18758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:24.348093 18758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:24.349275 18758 hybrid_clock.cc:648] HybridClock initialized: now 1790492544349273 us; error 40 us; skew 500 ppm
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:24.351987 18758 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:02:24.352424 18758 webserver.cc:460] Webserver started at http://127.3.57.1:37505/ using document root <none> and password file <none>
I20260927 07:02:24.352675 18758 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:24.352746 18758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:24.352892 18758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:24.353946 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "e0fa6b63a47145ea802b106ae315119e"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "7f9f6f115aa65ea268b731c704f4f101"
server_key_iv: "da3f57eb53336afa094bf0ddc940638c"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.354295 18758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "e0fa6b63a47145ea802b106ae315119e"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "7f9f6f115aa65ea268b731c704f4f101"
server_key_iv: "da3f57eb53336afa094bf0ddc940638c"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.355721 18758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:02:24.356712 18773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.357066 18758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:24.357169 18758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "e0fa6b63a47145ea802b106ae315119e"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "7f9f6f115aa65ea268b731c704f4f101"
server_key_iv: "da3f57eb53336afa094bf0ddc940638c"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.357337 18758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:24.377544 18758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:24.379415 18758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:24.379525 18758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:24.379734 18758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:24.380039 18758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:24.380072 18758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.380093 18758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:24.380143 18758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.385864 18758 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:41789
I20260927 07:02:24.385938 18886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:41789 every 8 connection(s)
I20260927 07:02:24.386242 18758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:02:24.392486 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18758
I20260927 07:02:24.392621 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:02:24.392844 3300 external_mini_cluster.cc:1428] Setting key 55b5453b708c7488429d1bed2ededb2b
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:24.397497 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.387062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:33953)
0927 07:02:24.387252 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.387288 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.388010 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.388311 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.388317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.389226 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:24.389439 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.389444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.390363 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.390364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.391241 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.391243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.391657 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.391658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.391776 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.392346 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:24.392354 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:24.394178 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:24.395909 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.395919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.395921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.396267 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.396634 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.396635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.396635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.396758 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.397052 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:24.397056 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:24.397160 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.397237 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.397348 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260927 07:02:24.397505 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.387098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:33953 (local address 127.3.57.62:37367)
0927 07:02:24.387142 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.387144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.387375 (+ 231us) server_negotiation.cc:407] Connection header received
0927 07:02:24.388158 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.388159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.388170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.388232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:24.389547 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.390278 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.391321 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.391529 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.391644 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.394278 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:24.394279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:24.394280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:24.394287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:24.395821 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.396347 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.396348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.396348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.396410 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.396835 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.396836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.396836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.396983 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:24.397056 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.397290 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.397391 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:02:24.398156 18887 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:24.398267 18887 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:24.398432 18887 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:24.398952 18696 ts_manager.cc:194] Registered new tserver with Master: e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789)
I20260927 07:02:24.399619 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:33953
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:02:24.419703 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:24.506152 18892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:24.506367 18892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:24.506392 18892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:24.507577 18892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:24.507615 18892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:24.507642 18892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:02:24.509120 18892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:24.509383 18892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:24.509621 18892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:24.515113 18898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.515146 18897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.515277 18900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:24.515381 18892 server_base.cc:1034] running on GCE node
I20260927 07:02:24.515641 18892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:24.515937 18892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:24.517134 18892 hybrid_clock.cc:648] HybridClock initialized: now 1790492544517115 us; error 37 us; skew 500 ppm
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:24.519678 18892 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:02:24.520052 18892 webserver.cc:460] Webserver started at http://127.3.57.2:37763/ using document root <none> and password file <none>
I20260927 07:02:24.520243 18892 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:24.520285 18892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:24.520375 18892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:24.521363 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "fa5b3510648a4f2daadba7f7190c4717"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "2fc47df93f1ecb3220aa22515b31a3ef"
server_key_iv: "ac36940fd41beb4db9db640c8641c96e"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.521657 18892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "fa5b3510648a4f2daadba7f7190c4717"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "2fc47df93f1ecb3220aa22515b31a3ef"
server_key_iv: "ac36940fd41beb4db9db640c8641c96e"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.522974 18892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:02:24.523890 18907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.524082 18892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:24.524178 18892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "fa5b3510648a4f2daadba7f7190c4717"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "2fc47df93f1ecb3220aa22515b31a3ef"
server_key_iv: "ac36940fd41beb4db9db640c8641c96e"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.524240 18892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:24.544950 18892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:24.546811 18892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:24.546919 18892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:24.547101 18892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:24.547430 18892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:24.547465 18892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.547487 18892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:24.547499 18892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.553259 18892 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:34685
I20260927 07:02:24.553355 19020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:34685 every 8 connection(s)
I20260927 07:02:24.553639 18892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:02:24.554956 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 18892
I20260927 07:02:24.555068 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:02:24.555269 3300 external_mini_cluster.cc:1428] Setting key 05ee57d31534e1180a80087b711b89c5
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:02:24.563771 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.554388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:37195 (local address 127.3.57.62:37367)
0927 07:02:24.554448 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.554450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.554652 (+ 202us) server_negotiation.cc:407] Connection header received
0927 07:02:24.555193 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.555195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.555208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.555264 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:24.556261 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.556751 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.557735 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.557927 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.558004 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.560508 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:24.560514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:24.560515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:24.560522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:24.562230 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.562680 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.562681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.562682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.562749 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.563007 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.563008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.563009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.563175 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:24.563265 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.563585 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.563672 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 07:02:24.563993 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.554372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:37195)
0927 07:02:24.554533 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.554568 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.555042 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.555332 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.555336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.555966 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:24.556155 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.556159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.556843 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.556844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.557645 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.557647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.558009 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.558010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.558130 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.558642 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:24.558650 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:24.560399 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:24.562338 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.562346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.562348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.562587 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.562821 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.562821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.562822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.562938 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.563255 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:24.563262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:24.563415 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.563525 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.563744 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260927 07:02:24.564839 19021 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:24.564968 19021 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:24.565147 19021 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:24.565544 18696 ts_manager.cc:194] Registered new tserver with Master: fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
I20260927 07:02:24.565973 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:37195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:02:24.581564 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:24.660946 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:24.661154 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:24.661180 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:24.662287 19026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:24.662323 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:24.662351 19026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:02:24.663723 19026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:24.663905 19026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:24.664077 19026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:24.669106 19031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.669130 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:24.669131 19032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:24.669165 19026 server_base.cc:1034] running on GCE node
I20260927 07:02:24.669483 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:24.669746 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:24.670909 19026 hybrid_clock.cc:648] HybridClock initialized: now 1790492544670907 us; error 39 us; skew 500 ppm
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:24.673326 19026 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:02:24.673789 19026 webserver.cc:460] Webserver started at http://127.3.57.3:33281/ using document root <none> and password file <none>
I20260927 07:02:24.674005 19026 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:24.674070 19026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:24.674216 19026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:02:24.675249 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "8cdb4ee216752552549a6367f7429832"
server_key_iv: "663f972bdebbf92749d568574c6c015f"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.675670 19026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "8cdb4ee216752552549a6367f7429832"
server_key_iv: "663f972bdebbf92749d568574c6c015f"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.676959 19026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:24.677654 19041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.677832 19026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:24.677929 19026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "8cdb4ee216752552549a6367f7429832"
server_key_iv: "663f972bdebbf92749d568574c6c015f"
server_key_version: "encryptionkey@0"
I20260927 07:02:24.678011 19026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:24.703083 19026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:24.704939 19026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:24.705090 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:24.705333 19026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:24.705619 19026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:24.705673 19026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.705709 19026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:24.705752 19026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:24.711314 19026 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:36737
I20260927 07:02:24.711385 19154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:36737 every 8 connection(s)
I20260927 07:02:24.711715 19026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:02:24.715955 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19026
I20260927 07:02:24.716060 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:02:24.716228 3300 external_mini_cluster.cc:1428] Setting key a6f164c83c5f0f787eb0494ddd68b218
Sep 27 07:02:24 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:24.720606 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.712438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:37011 (local address 127.3.57.62:37367)
0927 07:02:24.712515 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.712518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.712730 (+ 212us) server_negotiation.cc:407] Connection header received
0927 07:02:24.713247 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.713248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.713259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.713306 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:24.714219 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.714716 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.715707 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.715919 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.716002 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.718264 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:24.718266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:24.718266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:24.718272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:24.719403 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.719819 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.719820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.719820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.719869 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.720058 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.720059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.720059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.720160 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:24.720206 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.720483 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.720544 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:02:24.720887 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.712361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:37011)
0927 07:02:24.712569 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.712623 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.713140 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.713371 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.713374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.713940 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:24.714120 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.714124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.714812 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.714813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.715601 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.715603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.716012 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.716013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.716163 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.716686 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:24.716695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:24.718187 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:24.719471 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.719476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.719478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.719754 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.719929 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:24.719930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:24.719930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:24.720019 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:24.720271 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:24.720275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:24.720381 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.720456 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.720564 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260927 07:02:24.721312 19155 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:24.721432 19155 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:24.721608 19155 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:24.721974 18696 ts_manager.cc:194] Registered new tserver with Master: b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737)
I20260927 07:02:24.722287 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:37011
I20260927 07:02:24.727959 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:02:24.734730 18736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.729328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.3.57.62:37367)
0927 07:02:24.729383 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.729385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.729486 (+ 101us) server_negotiation.cc:407] Connection header received
0927 07:02:24.729524 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.729525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.729532 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.729586 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:24.730045 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.730537 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.731284 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.731456 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.731506 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.732639 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:24.732640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:24.732640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:24.732645 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:24.733740 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.734029 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.734029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.734030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.734072 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:24.734220 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:24.734228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:24.734228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:24.734446 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:24.734519 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.734598 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.734652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:02:24.736188 18696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:02:24.737696 18696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:02:24.743762 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.739854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.3.57.3:36737)
0927 07:02:24.740262 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.740265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.740273 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:24.740303 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.740305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.740318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.740610 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:24.740664 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.741677 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.742890 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.743337 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.743429 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.743483 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.743634 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.743670 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":72,"threads_started":1}
I20260927 07:02:24.743813 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.739781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34685 (local address 127.0.0.1:33396)
0927 07:02:24.740055 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.740066 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.740118 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.740398 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.740401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.740403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:24.740463 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.740465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.741419 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.741420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.742607 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.742609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.743541 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.743565 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.743643 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.743647 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.743692 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.743742 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260927 07:02:24.743883 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.739781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36737 (local address 127.0.0.1:58714)
0927 07:02:24.739923 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.739962 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.740027 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.740418 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.740419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.740421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:24.740476 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.740477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.741786 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.741787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.742821 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.742823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.743430 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.743431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.743514 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.743526 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.743620 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.743674 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":43,"threads_started":1}
I20260927 07:02:24.744105 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.739940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.3.57.2:34685)
0927 07:02:24.740247 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.740250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.740258 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:24.740287 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.740289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.740300 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.740386 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:24.740535 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.741319 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.742698 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.743399 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.743462 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.743509 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.743970 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.744009 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":68,"threads_started":1}
I20260927 07:02:24.744802 19089 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4672447b4494a9dffb4cd03ac602 (DEFAULT_TABLE table=test-table [id=a6470a5c293c4e3796ca829421db0dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:24.744801 18955 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4672447b4494a9dffb4cd03ac602 (DEFAULT_TABLE table=test-table [id=a6470a5c293c4e3796ca829421db0dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:24.745119 19089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4672447b4494a9dffb4cd03ac602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:24.745119 18955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4672447b4494a9dffb4cd03ac602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:24.745136 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.739955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.0.0.1:60742)
0927 07:02:24.740086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.740094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.740134 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.740457 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.740457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.740459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:24.740487 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.740488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.742906 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.742907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.743997 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.743999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.744903 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.744904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.744969 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.744973 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.745046 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.745087 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260927 07:02:24.745365 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.740085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.3.57.1:41789)
0927 07:02:24.740223 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.740228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.740248 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:02:24.740289 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.740291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.740306 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.740385 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:24.740563 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.741671 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.744076 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.744699 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.744792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.744847 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.745241 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.745286 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260927 07:02:24.745863 18821 tablet_service.cc:1467] Processing CreateTablet for tablet 110b4672447b4494a9dffb4cd03ac602 (DEFAULT_TABLE table=test-table [id=a6470a5c293c4e3796ca829421db0dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:24.746076 18821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110b4672447b4494a9dffb4cd03ac602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:24.748272 19173 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Bootstrap starting.
I20260927 07:02:24.748562 19174 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Bootstrap starting.
I20260927 07:02:24.748962 19173 tablet_bootstrap.cc:654] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:24.749209 19174 tablet_bootstrap.cc:654] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:24.749454 19173 log.cc:824] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:24.749756 19172 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Bootstrap starting.
I20260927 07:02:24.749796 19174 log.cc:824] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:24.750161 19173 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: No bootstrap required, opened a new log
I20260927 07:02:24.750250 19173 ts_tablet_manager.cc:1397] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:02:24.750365 19172 tablet_bootstrap.cc:654] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:24.750531 19174 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: No bootstrap required, opened a new log
I20260927 07:02:24.750613 19174 ts_tablet_manager.cc:1397] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:02:24.750916 19172 log.cc:824] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:24.751765 19172 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: No bootstrap required, opened a new log
I20260927 07:02:24.751854 19172 ts_tablet_manager.cc:1397] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:02:24.752269 19173 raft_consensus.cc:348] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.752444 19173 raft_consensus.cc:374] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:24.752481 19173 raft_consensus.cc:729] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e427a27f554ca1b9c2ee45066cedf6, State: Initialized, Role: FOLLOWER
I20260927 07:02:24.752521 19174 raft_consensus.cc:348] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.752619 19173 consensus_queue.cc:260] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.752686 19174 raft_consensus.cc:374] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:24.752719 19174 raft_consensus.cc:729] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Initialized, Role: FOLLOWER
I20260927 07:02:24.752887 19173 ts_tablet_manager.cc:1428] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:02:24.752871 19174 consensus_queue.cc:260] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.752995 19155 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
I20260927 07:02:24.753386 19174 ts_tablet_manager.cc:1428] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:02:24.753513 18887 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
I20260927 07:02:24.753561 19172 raft_consensus.cc:348] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.753738 19172 raft_consensus.cc:374] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:24.753785 19172 raft_consensus.cc:729] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5b3510648a4f2daadba7f7190c4717, State: Initialized, Role: FOLLOWER
I20260927 07:02:24.753897 19172 consensus_queue.cc:260] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.754151 19172 ts_tablet_manager.cc:1428] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:02:24.754191 19021 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
I20260927 07:02:24.761124 19180 raft_consensus.cc:482] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:24.761186 19180 raft_consensus.cc:504] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.761514 19180 leader_election.cc:288] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789)
I20260927 07:02:24.765051 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.761701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:51147 (local address 127.3.57.3:36737)
0927 07:02:24.761954 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.761955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.761960 (+ 5us) server_negotiation.cc:407] Connection header received
0927 07:02:24.761975 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.761975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.761982 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.762018 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:24.762168 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.762835 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764181 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.764730 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764798 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.764805 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.764963 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.764995 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260927 07:02:24.765051 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.761695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:59987 (local address 127.3.57.1:41789)
0927 07:02:24.761851 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.761852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.761858 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:02:24.761874 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.761874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.761882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.761919 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:24.762073 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.762792 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764181 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.764720 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764798 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.764805 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.764963 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.764995 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 07:02:24.765067 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.761664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.3.57.2:59987)
0927 07:02:24.761715 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.761726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.761775 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.761941 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.761942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.761943 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:24.761996 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.761997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.762863 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.762864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.764096 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.764099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.764809 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.764885 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.764886 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.764939 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.764984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":38}
I20260927 07:02:24.765074 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.761664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36737 (local address 127.3.57.2:51147)
0927 07:02:24.761814 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.761823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:24.761864 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:24.762038 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:24.762039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:24.762040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:24.762098 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.762099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.762899 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.762900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.764096 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:24.764099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.764809 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.764809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:24.764881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.764885 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:24.764939 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 07:02:24.764983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":20,"thread_start_us":78,"threads_started":1}
I20260927 07:02:24.765409 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4672447b4494a9dffb4cd03ac602" candidate_uuid: "fa5b3510648a4f2daadba7f7190c4717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa6b63a47145ea802b106ae315119e" is_pre_election: true
I20260927 07:02:24.765555 18841 raft_consensus.cc:2454] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5b3510648a4f2daadba7f7190c4717 in term 0.
I20260927 07:02:24.765542 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4672447b4494a9dffb4cd03ac602" candidate_uuid: "fa5b3510648a4f2daadba7f7190c4717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" is_pre_election: true
I20260927 07:02:24.765636 19109 raft_consensus.cc:2454] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5b3510648a4f2daadba7f7190c4717 in term 0.
I20260927 07:02:24.765797 18910 leader_election.cc:302] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0fa6b63a47145ea802b106ae315119e, fa5b3510648a4f2daadba7f7190c4717; no voters:
I20260927 07:02:24.765913 19180 raft_consensus.cc:2781] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:02:24.765990 19180 raft_consensus.cc:482] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:02:24.766022 19180 raft_consensus.cc:3037] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:24.766566 19180 raft_consensus.cc:504] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.766695 19180 leader_election.cc:288] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [CANDIDATE]: Term 1 election: Requested vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789)
I20260927 07:02:24.766799 19109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4672447b4494a9dffb4cd03ac602" candidate_uuid: "fa5b3510648a4f2daadba7f7190c4717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
I20260927 07:02:24.766832 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110b4672447b4494a9dffb4cd03ac602" candidate_uuid: "fa5b3510648a4f2daadba7f7190c4717" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0fa6b63a47145ea802b106ae315119e"
I20260927 07:02:24.766861 19109 raft_consensus.cc:3037] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:24.766908 18841 raft_consensus.cc:3037] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:24.767660 18841 raft_consensus.cc:2454] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5b3510648a4f2daadba7f7190c4717 in term 1.
I20260927 07:02:24.767660 19109 raft_consensus.cc:2454] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5b3510648a4f2daadba7f7190c4717 in term 1.
I20260927 07:02:24.767860 18910 leader_election.cc:302] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0fa6b63a47145ea802b106ae315119e, fa5b3510648a4f2daadba7f7190c4717; no voters:
I20260927 07:02:24.767951 19180 raft_consensus.cc:2781] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:24.768066 19180 raft_consensus.cc:686] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 1 LEADER]: Becoming Leader. State: Replica: fa5b3510648a4f2daadba7f7190c4717, State: Running, Role: LEADER
I20260927 07:02:24.768149 19180 consensus_queue.cc:237] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:24.768838 18696 catalog_manager.cc:5495] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "fa5b3510648a4f2daadba7f7190c4717" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } health_report { overall_health: UNKNOWN } } }
I20260927 07:02:24.778167 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.776278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.3.57.2:34685)
0927 07:02:24.776324 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:24.776326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:24.776333 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:02:24.776397 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:24.776398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:24.776412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:24.776458 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:24.776555 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.776964 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.777618 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:24.777785 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:24.777830 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:24.777951 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:24.778024 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:02:24.778077 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:24.778110 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 07:02:24.778124 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:02:24.779639 19109 raft_consensus.cc:1261] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Refusing update from remote peer fa5b3510648a4f2daadba7f7190c4717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:24.779680 18841 raft_consensus.cc:1261] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Refusing update from remote peer fa5b3510648a4f2daadba7f7190c4717: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:24.779964 19180 consensus_queue.cc:1035] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:24.780076 19182 consensus_queue.cc:1035] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:24.783702 19188 mvcc.cc:205] Tried to move back new op lower bound from 7333857463415042048 to 7333857463371026432. Current Snapshot: MvccSnapshot[applied={T|T < 7333857463415042048 or (T in {7333857463415042048})}]
I20260927 07:02:24.784425 18696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:24.784488 18696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:24.785089 18696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Sending DeleteTablet for 3 replicas of tablet 110b4672447b4494a9dffb4cd03ac602
I20260927 07:02:24.785380 19089 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4672447b4494a9dffb4cd03ac602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:24 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:58714
I20260927 07:02:24.785408 18821 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4672447b4494a9dffb4cd03ac602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:24 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60742
I20260927 07:02:24.785656 18955 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4672447b4494a9dffb4cd03ac602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:24 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33396
I20260927 07:02:24.785658 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18758
I20260927 07:02:24.786525 19196 tablet_replica.cc:330] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: stopping tablet replica
I20260927 07:02:24.786666 19196 raft_consensus.cc:2229] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:02:24.787190 19196 raft_consensus.cc:2258] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:24.787753 19196 ts_tablet_manager.cc:1905] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:24.789589 19196 ts_tablet_manager.cc:1918] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:24.789649 19196 log.cc:1196] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/110b4672447b4494a9dffb4cd03ac602
I20260927 07:02:24.789897 19196 ts_tablet_manager.cc:1939] T 110b4672447b4494a9dffb4cd03ac602 P fa5b3510648a4f2daadba7f7190c4717: Deleting consensus metadata
I20260927 07:02:24.791350 18682 catalog_manager.cc:4841] TS fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685): tablet 110b4672447b4494a9dffb4cd03ac602 (table test-table [id=a6470a5c293c4e3796ca829421db0dba]) successfully deleted
W20260927 07:02:24.793502 18682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:41789 (error 108)
I20260927 07:02:24.793988 19195 tablet_replica.cc:330] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: stopping tablet replica
I20260927 07:02:24.794087 19195 raft_consensus.cc:2229] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:24.794659 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18892
I20260927 07:02:24.796074 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:24.795985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.0.0.1:60744)
0927 07:02:24.796018 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:24.796048 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:41789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 07:02:24.796379 19195 raft_consensus.cc:2258] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:02:24.796470 18682 catalog_manager.cc:4584] TS e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 110b4672447b4494a9dffb4cd03ac602: Network error: Client connection negotiation failed: client connection to 127.3.57.1:41789: connect: Connection refused (error 111)
I20260927 07:02:24.796846 19195 ts_tablet_manager.cc:1905] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:24.798662 19195 ts_tablet_manager.cc:1918] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:24.798736 19195 log.cc:1196] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/110b4672447b4494a9dffb4cd03ac602
I20260927 07:02:24.798985 19195 ts_tablet_manager.cc:1939] T 110b4672447b4494a9dffb4cd03ac602 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting consensus metadata
I20260927 07:02:24.799678 18683 catalog_manager.cc:4841] TS b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737): tablet 110b4672447b4494a9dffb4cd03ac602 (table test-table [id=a6470a5c293c4e3796ca829421db0dba]) successfully deleted
I20260927 07:02:24.800583 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19026
I20260927 07:02:24.805792 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 18664
I20260927 07:02:44.811391 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--webserver_interface=127.3.57.62
--webserver_port=35359
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:02:44.894536 19198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:44.894708 19198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:44.894729 19198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:44.895807 19198 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:02:44.895838 19198 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:02:44.895851 19198 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:02:44.895862 19198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:02:44.895875 19198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:44.895888 19198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:44.895900 19198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:02:44.895918 19198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:02:44.897306 19198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=35359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:44.897472 19198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:44.897619 19198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:44.905913 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:44.906008 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:44.906015 19198 server_base.cc:1034] running on GCE node
W20260927 07:02:44.905920 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:44.906428 19198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:44.906756 19198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:44.907938 19198 hybrid_clock.cc:648] HybridClock initialized: now 1790492564907927 us; error 27 us; skew 500 ppm
Sep 27 07:02:44 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:44.915347 19198 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:02:44.915807 19198 webserver.cc:460] Webserver started at http://127.3.57.62:35359/ using document root <none> and password file <none>
I20260927 07:02:44.916079 19198 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:44.916131 19198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:44.917327 19198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:02:44.917889 19214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:44.918094 19198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:02:44.918171 19198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "1a8f5d35938a4d0b99891ed240de11c2"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "488b1171be9483738c8a5432ae612460"
server_key_iv: "56286f54a335088412d8e2e30eae9d18"
server_key_version: "encryptionkey@0"
I20260927 07:02:44.918517 19198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:44.940251 19198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:44.942277 19198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:44.942380 19198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:44.946017 19198 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37367
I20260927 07:02:44.946098 19266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37367 every 8 connection(s)
I20260927 07:02:44.946437 19198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:02:44.947909 19267 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:02:44.948534 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap starting.
I20260927 07:02:44.956462 19267 log.cc:824] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:44.956461 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19198
I20260927 07:02:44.956739 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:41789
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=37505
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:44.958216 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:44.958496 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap complete.
I20260927 07:02:44.960819 19267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:44.961058 19267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Initialized, Role: FOLLOWER
I20260927 07:02:44.961236 19267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:44.961311 19267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:02:44.961374 19267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:02:44.961443 19267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:02:44.962374 19267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:44.962491 19267 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8f5d35938a4d0b99891ed240de11c2; no voters:
I20260927 07:02:44.962700 19267 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:02:44.962728 19271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:02:44.962841 19271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 LEADER]: Becoming Leader. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Running, Role: LEADER
I20260927 07:02:44.962953 19267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:02:44.962930 19271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:02:44.963284 19272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:02:44.963315 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8f5d35938a4d0b99891ed240de11c2. Latest consensus state: current_term: 2 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:02:44.963464 19272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:44.963480 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:02:44.963873 19280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:02:44.964679 19280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:02:44.964797 19280 catalog_manager.cc:1259] Loaded cluster ID: cb5d9e9cd4024881b59e6c2689868e4a
I20260927 07:02:44.965062 19280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:02:44.965845 19280 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:02:44.966039 19280 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Loaded TSK: 0
I20260927 07:02:44.976796 19280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Generated new TSK 1
I20260927 07:02:44.976907 19280 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:02:45.051406 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:45.051607 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:45.051631 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:45.052755 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:45.052793 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:45.052845 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:02:45.054260 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:41789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=37505
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:45.054472 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:45.054701 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:45.059665 19294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:45.059747 19295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:45.059826 19297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:45.059929 19270 server_base.cc:1034] running on GCE node
I20260927 07:02:45.060163 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:45.060428 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:45.061582 19270 hybrid_clock.cc:648] HybridClock initialized: now 1790492565061571 us; error 26 us; skew 500 ppm
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:45.064491 19270 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:02:45.064988 19270 webserver.cc:460] Webserver started at http://127.3.57.1:37505/ using document root <none> and password file <none>
I20260927 07:02:45.065234 19270 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:45.065311 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:45.066942 19270 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:02:45.067626 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.067808 19270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.067910 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "e0fa6b63a47145ea802b106ae315119e"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "7f9f6f115aa65ea268b731c704f4f101"
server_key_iv: "da3f57eb53336afa094bf0ddc940638c"
server_key_version: "encryptionkey@0"
I20260927 07:02:45.068279 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:45.096558 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:45.098834 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:45.098964 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:45.099177 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:45.099632 19311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:02:45.101018 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:02:45.101074 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:02:45.101105 19270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:02:45.101900 19270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:02:45.101963 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:45.102032 19311 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Bootstrap starting.
I20260927 07:02:45.108771 19270 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:41789
I20260927 07:02:45.108980 19418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:41789 every 8 connection(s)
I20260927 07:02:45.109215 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:02:45.111781 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19270
I20260927 07:02:45.112087 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:34685
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=37763
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:45.113317 19311 log.cc:824] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:45.115715 19311 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:02:45.116044 19311 tablet_bootstrap.cc:492] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Bootstrap complete.
I20260927 07:02:45.117130 19311 ts_tablet_manager.cc:1397] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260927 07:02:45.118075 19311 raft_consensus.cc:348] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.118409 19311 raft_consensus.cc:729] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Initialized, Role: FOLLOWER
I20260927 07:02:45.118541 19311 consensus_queue.cc:260] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:45.118791 19311 ts_tablet_manager.cc:1428] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 07:02:45.123003 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.109893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:47311)
0927 07:02:45.110072 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.110118 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.110660 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.112167 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.112171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.112492 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:45.112731 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.112735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.114606 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.114607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.115826 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.115829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.116695 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.116696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.116832 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.117374 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:45.117383 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:45.119046 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:45.121461 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.121466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.121468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.121660 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.121965 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.121965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.121966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.122135 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.122583 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:45.122588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:45.122688 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.122765 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.122874 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260927 07:02:45.123206 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.110023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:47311 (local address 127.3.57.62:37367)
0927 07:02:45.111696 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.111705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.111744 (+ 39us) server_negotiation.cc:407] Connection header received
0927 07:02:45.111910 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.111917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.111976 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.112110 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.113014 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.114470 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.115910 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.116579 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.116685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.119153 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.119157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.119157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.119169 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.121384 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.121743 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.121746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.121746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.121873 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.122199 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.122200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.122200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.122506 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.122603 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.122860 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.123013 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":68,"threads_started":1}
I20260927 07:02:45.124094 19419 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:45.124194 19419 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:45.124379 19419 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:45.125195 19231 ts_manager.cc:194] Registered new tserver with Master: e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789)
I20260927 07:02:45.126479 19231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:47311
I20260927 07:02:45.127460 19419 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 07:02:45.134140 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.126780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.3.57.1:41789)
0927 07:02:45.126990 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.126993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.127001 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:45.127026 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.127028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.127038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.127118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.127615 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.128341 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.129117 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.129478 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.129580 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.131520 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.131527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.131527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.131533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.132929 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.133311 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.133313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.133314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.133368 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.133576 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.133577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.133577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.133697 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.133787 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.133973 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.134049 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260927 07:02:45.134141 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.125812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.0.0.1:55464)
0927 07:02:45.125964 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.125997 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.126054 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.127121 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.127127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.127423 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:45.127462 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.127464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.128421 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.128422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.129023 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.129025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.129579 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.129583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.129655 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.130074 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:45.130078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:45.131435 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:45.133029 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.133035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.133037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.133222 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.133440 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.133441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.133441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.133497 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.133768 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:45.133774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:45.133823 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.133935 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.134050 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260927 07:02:45.134491 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet 110b4672447b4494a9dffb4cd03ac602 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:24 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:55464
I20260927 07:02:45.134938 19428 tablet_replica.cc:330] stopping tablet replica
I20260927 07:02:45.135046 19428 raft_consensus.cc:2229] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:45.135104 19428 raft_consensus.cc:2258] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:45.135380 19428 ts_tablet_manager.cc:1905] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:45.138017 19428 ts_tablet_manager.cc:1918] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:45.138092 19428 log.cc:1196] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal/wals/110b4672447b4494a9dffb4cd03ac602
I20260927 07:02:45.138314 19428 ts_tablet_manager.cc:1939] T 110b4672447b4494a9dffb4cd03ac602 P e0fa6b63a47145ea802b106ae315119e: Deleting consensus metadata
I20260927 07:02:45.138784 19217 catalog_manager.cc:4841] TS e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789): tablet 110b4672447b4494a9dffb4cd03ac602 (table test-table [id=a6470a5c293c4e3796ca829421db0dba]) successfully deleted
W20260927 07:02:45.198761 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:45.198941 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:45.198963 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:45.200129 19423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:45.200166 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:45.200191 19423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:02:45.201562 19423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:34685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=37763
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:45.201782 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:45.202004 19423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:45.206801 19436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:45.206815 19434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:45.206972 19423 server_base.cc:1034] running on GCE node
W20260927 07:02:45.207022 19433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:45.207255 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:45.207491 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:45.208647 19423 hybrid_clock.cc:648] HybridClock initialized: now 1790492565208656 us; error 50 us; skew 500 ppm
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:45.210963 19423 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:02:45.211369 19423 webserver.cc:460] Webserver started at http://127.3.57.2:37763/ using document root <none> and password file <none>
I20260927 07:02:45.211645 19423 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:45.211724 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:45.213009 19423 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:02:45.213547 19443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.213708 19423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:02:45.213802 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "fa5b3510648a4f2daadba7f7190c4717"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "2fc47df93f1ecb3220aa22515b31a3ef"
server_key_iv: "ac36940fd41beb4db9db640c8641c96e"
server_key_version: "encryptionkey@0"
I20260927 07:02:45.214121 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:45.232134 19423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:45.234014 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:45.234171 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:45.234397 19423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:45.234704 19423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:45.234757 19423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.234797 19423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:45.234826 19423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.240623 19423 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:34685
I20260927 07:02:45.240671 19556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:34685 every 8 connection(s)
I20260927 07:02:45.240967 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:45.248445 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19423
I20260927 07:02:45.248710 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:36737
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=33281
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:02:45.250036 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.241614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:56057 (local address 127.3.57.62:37367)
0927 07:02:45.241671 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.241672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.241868 (+ 196us) server_negotiation.cc:407] Connection header received
0927 07:02:45.242409 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.242411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.242426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.242481 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.243411 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.243957 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.244947 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.245156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.245239 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.247323 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.247325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.247325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.247332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.248614 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.249097 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.249098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.249099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.249156 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.249405 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.249406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.249406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.249519 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.249595 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.249857 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.249945 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 07:02:45.250204 19559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.241601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:56057)
0927 07:02:45.241756 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.241787 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.242294 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.242541 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.242545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.243121 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:45.243301 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.243305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.244047 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.244048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.244805 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.244807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.245249 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.245250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.245372 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.245862 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:45.245870 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:45.247225 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:45.248699 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.248705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.248707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.249014 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.249230 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.249230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.249231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.249336 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.249587 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:45.249593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:45.249726 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.249825 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.249981 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260927 07:02:45.250663 19557 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:45.250761 19557 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:45.250946 19557 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:45.251350 19231 ts_manager.cc:194] Registered new tserver with Master: fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
I20260927 07:02:45.251794 19231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:56057
W20260927 07:02:45.327348 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:02:45.327558 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:02:45.327612 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:02:45.328790 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:02:45.328850 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:02:45.328896 19560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:02:45.330312 19560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:36737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=33281
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:02:45.330566 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:02:45.330785 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:02:45.335583 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:45.335572 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:02:45.335567 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:02:45.335696 19560 server_base.cc:1034] running on GCE node
I20260927 07:02:45.336037 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:02:45.336277 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:02:45.337455 19560 hybrid_clock.cc:648] HybridClock initialized: now 1790492565337447 us; error 38 us; skew 500 ppm
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:02:45.339847 19560 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:02:45.340307 19560 webserver.cc:460] Webserver started at http://127.3.57.3:33281/ using document root <none> and password file <none>
I20260927 07:02:45.340524 19560 fs_manager.cc:359] Metadata directory not provided
I20260927 07:02:45.340601 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:02:45.341884 19560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:02:45.342419 19575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.342573 19560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.342665 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "8cdb4ee216752552549a6367f7429832"
server_key_iv: "663f972bdebbf92749d568574c6c015f"
server_key_version: "encryptionkey@0"
I20260927 07:02:45.342993 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:02:45.371352 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:02:45.373554 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:02:45.373716 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:02:45.373965 19560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:02:45.374270 19560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:02:45.374325 19560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.374367 19560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:02:45.374392 19560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:02:45.380434 19560 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:36737
I20260927 07:02:45.380486 19688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:36737 every 8 connection(s)
I20260927 07:02:45.380810 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:02:45.383718 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19560
Sep 27 07:02:45 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:02:45.390869 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.384140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.3.57.62:37367)
0927 07:02:45.384238 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.384240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.384358 (+ 118us) server_negotiation.cc:407] Connection header received
0927 07:02:45.384413 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.384413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.384424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.384501 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.385072 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.385723 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.386546 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.386767 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.386819 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.388173 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.388174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.388174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.388180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.389758 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.390136 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.390137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.390137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.390178 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.390363 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.390363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.390363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.390458 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.390612 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.390680 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.390741 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
I20260927 07:02:45.391013 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.381471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:43581 (local address 127.3.57.62:37367)
0927 07:02:45.381540 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.381543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.381805 (+ 262us) server_negotiation.cc:407] Connection header received
0927 07:02:45.382300 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.382301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.382312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.382365 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.383265 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.383874 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.385100 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.385281 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.385328 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.387680 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.387682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.387682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.387689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.389518 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.389941 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.389941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.389941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.389991 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.390278 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.390279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.390280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.390428 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.390517 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.390843 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.390913 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:02:45.391345 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.381453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:43581)
0927 07:02:45.381643 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.381680 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.382215 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.382441 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.382444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.383003 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:02:45.383190 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.383194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.383979 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.383980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.385000 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.385003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.385357 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.385358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.385495 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.386057 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:02:45.386065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:02:45.387587 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:02:45.389605 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.389610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.389612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.389865 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.390069 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:02:45.390070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:02:45.390071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:02:45.390193 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:02:45.390529 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:02:45.390535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:02:45.390690 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.390799 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.390989 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260927 07:02:45.391825 19689 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37367
I20260927 07:02:45.391937 19689 heartbeater.cc:464] Registering TS with master...
I20260927 07:02:45.392078 19689 heartbeater.cc:511] Master 127.3.57.62:37367 requested a full tablet report, sending...
I20260927 07:02:45.392426 19231 ts_manager.cc:194] Registered new tserver with Master: b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737)
I20260927 07:02:45.392755 19231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:43581
I20260927 07:02:45.402792 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:02:45.405576 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.403131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.3.57.62:37367)
0927 07:02:45.403196 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.403198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.403288 (+ 90us) server_negotiation.cc:407] Connection header received
0927 07:02:45.403362 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.403363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.403400 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.403474 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:45.403623 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.404099 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.404861 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.405069 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.405146 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.405310 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:45.405354 (+ 44us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 07:02:45.405506 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44462: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 07:02:45.405632 19693 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44462: token expired
W20260927 07:02:45.405712 19158 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.57.62:37367)
I20260927 07:02:45.411615 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.405928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.3.57.62:37367)
0927 07:02:45.405959 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.405961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.405969 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:02:45.406032 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.406032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.406039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.406087 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:02:45.406515 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.406921 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.407681 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.407843 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.407893 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.408856 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:02:45.408857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:02:45.408857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:02:45.408863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:02:45.410204 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.410611 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.410612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.410612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.410672 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:02:45.410925 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:02:45.410926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:02:45.410926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:02:45.411122 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:02:45.411227 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.411399 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.411473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 07:02:45.412863 19231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:02:45.414710 19231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:02:45.416919 19353 tablet_service.cc:1467] Processing CreateTablet for tablet 97cda1bad9b74016a2693147dab8ee76 (DEFAULT_TABLE table=test-table [id=273f399773f94ee184551196de1b4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:45.417121 19353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cda1bad9b74016a2693147dab8ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:45.418550 19311 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Bootstrap starting.
I20260927 07:02:45.418854 19311 tablet_bootstrap.cc:654] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:45.419391 19311 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: No bootstrap required, opened a new log
I20260927 07:02:45.419449 19311 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:45.419811 19311 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.419921 19311 raft_consensus.cc:374] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:45.419975 19311 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Initialized, Role: FOLLOWER
I20260927 07:02:45.420032 19311 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.420132 19311 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 07:02:45.420492 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.416492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34685 (local address 127.0.0.1:53596)
0927 07:02:45.416719 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.416729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.416780 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.417120 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.417120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.417122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:45.417161 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.417162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.418487 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.418488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.419363 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.419366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.420217 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.420218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.420283 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.420319 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.420377 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.420426 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":9,"thread_start_us":108,"threads_started":1}
I20260927 07:02:45.420512 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.416780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.3.57.2:34685)
0927 07:02:45.416936 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.416940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.416949 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:02:45.416985 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.416987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.417024 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.417231 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:45.417325 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.418392 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.419472 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.420097 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.420211 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.420271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.420399 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.420438 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260927 07:02:45.421274 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.416508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.3.57.3:36737)
0927 07:02:45.416668 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.416673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.416682 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:02:45.416718 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.416720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.416746 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.416956 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:45.417034 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.418036 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.420202 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.420970 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.421042 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.421092 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.421190 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.421220 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260927 07:02:45.421280 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.416444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36737 (local address 127.0.0.1:33766)
0927 07:02:45.416508 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.416519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.416558 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.416850 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.416851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.416852 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:45.416888 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.416889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.418107 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.418108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.419150 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.419153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.421053 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.421054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.421117 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.421121 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.421182 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.421225 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 07:02:45.421195 19490 tablet_service.cc:1467] Processing CreateTablet for tablet 97cda1bad9b74016a2693147dab8ee76 (DEFAULT_TABLE table=test-table [id=273f399773f94ee184551196de1b4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:45.421526 19490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cda1bad9b74016a2693147dab8ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:45.421962 19623 tablet_service.cc:1467] Processing CreateTablet for tablet 97cda1bad9b74016a2693147dab8ee76 (DEFAULT_TABLE table=test-table [id=273f399773f94ee184551196de1b4a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:02:45.422247 19623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cda1bad9b74016a2693147dab8ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:02:45.425020 19699 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Bootstrap starting.
I20260927 07:02:45.425595 19700 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Bootstrap starting.
I20260927 07:02:45.425661 19699 tablet_bootstrap.cc:654] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:45.426277 19700 tablet_bootstrap.cc:654] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Neither blocks nor log segments found. Creating new log.
I20260927 07:02:45.426278 19699 log.cc:824] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:45.426852 19700 log.cc:824] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:02:45.427011 19699 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: No bootstrap required, opened a new log
I20260927 07:02:45.427089 19699 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:02:45.427644 19700 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: No bootstrap required, opened a new log
I20260927 07:02:45.427709 19700 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:02:45.429140 19699 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.429311 19699 raft_consensus.cc:374] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:45.429345 19699 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5b3510648a4f2daadba7f7190c4717, State: Initialized, Role: FOLLOWER
I20260927 07:02:45.429471 19699 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.429749 19699 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:02:45.429780 19557 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
I20260927 07:02:45.429756 19700 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.429910 19700 raft_consensus.cc:374] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:02:45.429955 19700 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e427a27f554ca1b9c2ee45066cedf6, State: Initialized, Role: FOLLOWER
I20260927 07:02:45.430080 19700 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.430361 19700 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 07:02:45.430377 19689 heartbeater.cc:503] Master 127.3.57.62:37367 was elected leader, sending a full tablet report...
W20260927 07:02:45.491496 19558 tablet.cc:2369] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:02:45.612085 19420 tablet.cc:2369] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:02:45.631366 19690 tablet.cc:2369] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:02:45.666728 19425 raft_consensus.cc:482] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:02:45.666874 19425 raft_consensus.cc:504] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.667253 19425 leader_election.cc:288] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
I20260927 07:02:45.671279 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.667524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:53211 (local address 127.3.57.3:36737)
0927 07:02:45.667582 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.667584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.667762 (+ 178us) server_negotiation.cc:407] Connection header received
0927 07:02:45.667807 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.667808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.667819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.667877 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:45.668065 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.668802 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670297 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.670881 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670957 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.670967 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.671176 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.671210 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:02:45.671280 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.667512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:42737 (local address 127.3.57.2:34685)
0927 07:02:45.667576 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.667578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.667649 (+ 71us) server_negotiation.cc:407] Connection header received
0927 07:02:45.667785 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.667786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.667799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.667877 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:02:45.668064 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.668803 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670297 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.670881 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670957 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.670967 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.671176 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.671210 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:02:45.671346 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.667437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34685 (local address 127.3.57.1:42737)
0927 07:02:45.667576 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.667586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.667689 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.667892 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.667893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.667895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:45.667980 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.667982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.668918 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.668919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.670208 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.670212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.670976 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.671067 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.671099 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.671159 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.671219 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":95,"thread_start_us":70,"threads_started":1}
I20260927 07:02:45.671346 19705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.667437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36737 (local address 127.3.57.1:53211)
0927 07:02:45.667689 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.667699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:02:45.667752 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:02:45.667893 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:02:45.667894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:02:45.667897 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:02:45.667993 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.667994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.668912 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.668913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.670210 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:02:45.670212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.670974 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.670975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:02:45.671067 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.671097 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 07:02:45.671159 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 07:02:45.671219 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":59,"thread_start_us":57,"threads_started":1}
I20260927 07:02:45.671744 19511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5b3510648a4f2daadba7f7190c4717" is_pre_election: true
I20260927 07:02:45.671751 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" is_pre_election: true
I20260927 07:02:45.671893 19511 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 0.
I20260927 07:02:45.671890 19643 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 0.
I20260927 07:02:45.672128 19307 leader_election.cc:302] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0fa6b63a47145ea802b106ae315119e, fa5b3510648a4f2daadba7f7190c4717; no voters:
I20260927 07:02:45.672324 19425 raft_consensus.cc:2781] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:02:45.672374 19425 raft_consensus.cc:482] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:02:45.672389 19425 raft_consensus.cc:3037] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:45.673065 19425 raft_consensus.cc:504] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.673199 19425 leader_election.cc:288] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 1 election: Requested vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
I20260927 07:02:45.673341 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
I20260927 07:02:45.673427 19643 raft_consensus.cc:3037] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:45.674067 19643 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 1.
I20260927 07:02:45.674191 19511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa5b3510648a4f2daadba7f7190c4717"
I20260927 07:02:45.674271 19308 leader_election.cc:302] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1e427a27f554ca1b9c2ee45066cedf6, e0fa6b63a47145ea802b106ae315119e; no voters:
I20260927 07:02:45.674296 19511 raft_consensus.cc:3037] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:02:45.674343 19425 raft_consensus.cc:2781] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:02:45.674453 19425 raft_consensus.cc:686] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 LEADER]: Becoming Leader. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Running, Role: LEADER
I20260927 07:02:45.674522 19425 consensus_queue.cc:237] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:02:45.675204 19511 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 1.
I20260927 07:02:45.675216 19231 catalog_manager.cc:5495] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e reported cstate change: term changed from 0 to 1, leader changed from <none> to e0fa6b63a47145ea802b106ae315119e (127.3.57.1). New cstate: current_term: 1 leader_uuid: "e0fa6b63a47145ea802b106ae315119e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } health_report { overall_health: HEALTHY } } }
I20260927 07:02:45.701643 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.699455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.3.57.1:41789)
0927 07:02:45.699601 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 07:02:45.699603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:02:45.699609 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:02:45.699641 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:02:45.699642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:02:45.699657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:02:45.699716 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:02:45.699843 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.700309 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.700964 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:02:45.701141 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:02:45.701187 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:02:45.701300 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:02:45.701391 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:02:45.701440 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:02:45.701476 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 07:02:45.701492 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260927 07:02:45.703133 19511 raft_consensus.cc:1261] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Refusing update from remote peer e0fa6b63a47145ea802b106ae315119e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:45.703137 19643 raft_consensus.cc:1261] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Refusing update from remote peer e0fa6b63a47145ea802b106ae315119e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:02:45.703465 19707 consensus_queue.cc:1035] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:45.703637 19425 consensus_queue.cc:1035] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:02:45.706516 19711 mvcc.cc:205] Tried to move back new op lower bound from 7333857549117546496 to 7333857549003354112. Current Snapshot: MvccSnapshot[applied={T|T < 7333857549117546496 or (T in {7333857549117546496})}]
I20260927 07:02:45.706653 19712 mvcc.cc:205] Tried to move back new op lower bound from 7333857549117546496 to 7333857549003354112. Current Snapshot: MvccSnapshot[applied={T|T < 7333857549117546496}]
I20260927 07:02:45.708001 19230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:45.708112 19230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:02:45.708765 19230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Sending DeleteTablet for 3 replicas of tablet 97cda1bad9b74016a2693147dab8ee76
I20260927 07:02:45.709067 19353 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cda1bad9b74016a2693147dab8ee76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:45 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:55464
I20260927 07:02:45.709064 19490 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cda1bad9b74016a2693147dab8ee76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:45 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:53596
I20260927 07:02:45.709179 19623 tablet_service.cc:1514] Processing DeleteTablet for tablet 97cda1bad9b74016a2693147dab8ee76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:02:45 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:33766
I20260927 07:02:45.709304 19722 tablet_replica.cc:330] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: stopping tablet replica
I20260927 07:02:45.709389 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19270
I20260927 07:02:45.709380 19723 tablet_replica.cc:330] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: stopping tablet replica
I20260927 07:02:45.709400 19722 raft_consensus.cc:2229] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:02:45.709484 19723 raft_consensus.cc:2229] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:45.709583 19723 raft_consensus.cc:2258] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:45.709594 19722 raft_consensus.cc:2258] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:45.710047 19723 ts_tablet_manager.cc:1905] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:02:45.711491 19723 ts_tablet_manager.cc:1918] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:02:45.711587 19723 log.cc:1196] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/97cda1bad9b74016a2693147dab8ee76
I20260927 07:02:45.711863 19723 ts_tablet_manager.cc:1939] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Deleting consensus metadata
I20260927 07:02:45.712231 19218 catalog_manager.cc:4841] TS b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737): tablet 97cda1bad9b74016a2693147dab8ee76 (table test-table [id=273f399773f94ee184551196de1b4a5b]) successfully deleted
W20260927 07:02:45.715148 19217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:41789 (error 108)
I20260927 07:02:45.715426 19721 tablet_replica.cc:330] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: stopping tablet replica
I20260927 07:02:45.715536 19721 raft_consensus.cc:2229] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:02:45.715617 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.715467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.0.0.1:49900)
0927 07:02:45.715528 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.715585 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:41789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:02:45.715660 19721 raft_consensus.cc:2258] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:02:45.715867 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19423
W20260927 07:02:45.715732 19217 catalog_manager.cc:4584] TS e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97cda1bad9b74016a2693147dab8ee76: Network error: Client connection negotiation failed: client connection to 127.3.57.1:41789: connect: Connection refused (error 111)
I20260927 07:02:45.721415 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:02:45.721306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34685 (local address 127.0.0.1:46768)
0927 07:02:45.721360 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:02:45.721384 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:34685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:02:45.721416 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19560
I20260927 07:02:45.726615 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19198
I20260927 07:03:05.732276 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--webserver_interface=127.3.57.62
--webserver_port=35359
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:05.813838 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:05.814031 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:05.814054 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:05.815168 19725 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:03:05.815204 19725 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:03:05.815217 19725 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:03:05.815228 19725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:05.815287 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:05.815315 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:05.815339 19725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:05.815359 19725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:05.816825 19725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=35359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:05.817071 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:05.817294 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:05.822415 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:05.822476 19725 server_base.cc:1034] running on GCE node
W20260927 07:03:05.822432 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:05.822504 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:05.822973 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:05.823261 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:05.824416 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790492585824407 us; error 43 us; skew 500 ppm
Sep 27 07:03:05 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:05.833540 19725 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:05.833968 19725 webserver.cc:460] Webserver started at http://127.3.57.62:35359/ using document root <none> and password file <none>
I20260927 07:03:05.834214 19725 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:05.834261 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:05.835675 19725 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:03:05.836408 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:05.836622 19725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:05.836741 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "1a8f5d35938a4d0b99891ed240de11c2"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "488b1171be9483738c8a5432ae612460"
server_key_iv: "56286f54a335088412d8e2e30eae9d18"
server_key_version: "encryptionkey@0"
I20260927 07:03:05.837105 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:05.847357 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:05.849265 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:05.849407 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:05.853315 19725 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37367
I20260927 07:03:05.853363 19792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37367 every 8 connection(s)
I20260927 07:03:05.853727 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:05.855093 19793 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:03:05.855790 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap starting.
I20260927 07:03:05.857106 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19725
I20260927 07:03:05.857394 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:41789
--local_ip_for_outbound_sockets=127.3.57.1
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=37505
--webserver_interface=127.3.57.1
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:05.864964 19793 log.cc:824] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:05.867347 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:03:05.867702 19793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Bootstrap complete.
I20260927 07:03:05.870237 19793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:03:05.870451 19793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Initialized, Role: FOLLOWER
I20260927 07:03:05.870618 19793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:03:05.870676 19793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:05.870738 19793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:05.870788 19793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:03:05.871830 19793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:03:05.872037 19793 leader_election.cc:302] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a8f5d35938a4d0b99891ed240de11c2; no voters:
I20260927 07:03:05.872280 19793 leader_election.cc:288] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:03:05.872321 19797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:03:05.872464 19797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [term 3 LEADER]: Becoming Leader. State: Replica: 1a8f5d35938a4d0b99891ed240de11c2, State: Running, Role: LEADER
I20260927 07:03:05.872550 19793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:05.872550 19797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } }
I20260927 07:03:05.872901 19797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a8f5d35938a4d0b99891ed240de11c2. Latest consensus state: current_term: 3 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:03:05.872984 19797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:05.872905 19799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1a8f5d35938a4d0b99891ed240de11c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a8f5d35938a4d0b99891ed240de11c2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37367 } } }
I20260927 07:03:05.873168 19799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:05.873180 19803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:05.874090 19803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:05.874235 19803 catalog_manager.cc:1259] Loaded cluster ID: cb5d9e9cd4024881b59e6c2689868e4a
I20260927 07:03:05.874456 19803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:05.875689 19803 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:05.875885 19803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Loaded TSK: 0
I20260927 07:03:05.875932 19803 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a8f5d35938a4d0b99891ed240de11c2: Loaded TSK: 1
F20260927 07:03:05.876168 19803 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdd039c29ad google::LogMessage::Fail() at ??:0
@ 0x7fdd039c588a google::LogMessage::SendToLog() at ??:0
@ 0x7fdd039c24de google::LogMessage::Flush() at ??:0
@ 0x7fdd039c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdd0612f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdd0612cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdd0b91c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdd0b91c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdd0b91c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fdd0b8e2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdd0b8ecad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdd058900d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdd05885954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdd066e96db start_thread at ??:0
@ 0x7fdd024c061f clone at ??:0
W20260927 07:03:05.953858 19795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:05.954092 19795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:05.954133 19795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:05.955273 19795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:05.955338 19795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:05.955386 19795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:05.956899 19795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:41789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=37505
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:05.957374 19795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:05.957764 19795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:05.962742 19837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:05.962738 19835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:05.962736 19834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:05.962847 19795 server_base.cc:1034] running on GCE node
I20260927 07:03:05.963131 19795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:05.963385 19795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:05.964577 19795 hybrid_clock.cc:648] HybridClock initialized: now 1790492585964560 us; error 31 us; skew 500 ppm
Sep 27 07:03:05 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:05.967303 19795 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:03:05.967876 19795 webserver.cc:460] Webserver started at http://127.3.57.1:37505/ using document root <none> and password file <none>
I20260927 07:03:05.968127 19795 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:05.968192 19795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:05.969905 19795 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:05.970686 19844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:05.970906 19795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:05.970995 19795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "e0fa6b63a47145ea802b106ae315119e"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "7f9f6f115aa65ea268b731c704f4f101"
server_key_iv: "da3f57eb53336afa094bf0ddc940638c"
server_key_version: "encryptionkey@0"
I20260927 07:03:05.971422 19795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:05.991178 19795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:05.993793 19795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:05.993947 19795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:05.994211 19795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:05.994756 19851 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:03:05.996300 19795 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:03:05.996361 19795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:03:05.996399 19795 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:03:05.997269 19795 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:03:05.997315 19795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:05.997385 19851 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Bootstrap starting.
I20260927 07:03:06.005486 19795 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:41789
I20260927 07:03:06.005578 19958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:41789 every 8 connection(s)
I20260927 07:03:06.005942 19795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:06.008482 19851 log.cc:824] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:06.010622 19851 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:03:06.010910 19851 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Bootstrap complete.
I20260927 07:03:06.011886 19851 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260927 07:03:06.012910 19851 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:06.013206 19851 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Initialized, Role: FOLLOWER
I20260927 07:03:06.013298 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19795
I20260927 07:03:06.013396 19851 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:06.013623 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:34685
--local_ip_for_outbound_sockets=127.3.57.2
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=37763
--webserver_interface=127.3.57.2
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:06.013710 19851 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260927 07:03:06.084069 19961 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:06.006708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:35403)
0927 07:03:06.006945 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.007000 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:06.007788 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:06.083862 (+ 76074us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":98,"threads_started":1}
W20260927 07:03:06.084342 19959 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37367 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:37367: Client connection negotiation failed: client connection to 127.3.57.62:37367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:03:06.084718 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.084583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:40781)
0927 07:03:06.084642 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.084690 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:03:06.085110 19961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.085010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:56357)
0927 07:03:06.085059 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.085088 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:03:06.085237 19959 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:03:06.097172 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:06.097358 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:06.097381 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:06.098512 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:06.098554 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:06.098582 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:06.100055 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:34685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=37763
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:06.100281 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:06.100513 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:06.105389 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:06.105402 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:06.105405 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:06.105559 19964 server_base.cc:1034] running on GCE node
I20260927 07:03:06.105867 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:06.106145 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:06.107301 19964 hybrid_clock.cc:648] HybridClock initialized: now 1790492586107288 us; error 47 us; skew 500 ppm
Sep 27 07:03:06 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:06.109781 19964 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:03:06.110239 19964 webserver.cc:460] Webserver started at http://127.3.57.2:37763/ using document root <none> and password file <none>
I20260927 07:03:06.110456 19964 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:06.110500 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:06.111851 19964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:06.112490 19979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:06.112701 19964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:06.112771 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "fa5b3510648a4f2daadba7f7190c4717"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "2fc47df93f1ecb3220aa22515b31a3ef"
server_key_iv: "ac36940fd41beb4db9db640c8641c96e"
server_key_version: "encryptionkey@0"
I20260927 07:03:06.113119 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:06.130056 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:06.132828 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:06.133016 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:06.133251 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:06.133780 19986 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 07:03:06.135159 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:03:06.135231 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 07:03:06.135275 19964 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:03:06.135941 19964 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:03:06.135998 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:06.136063 19986 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Bootstrap starting.
I20260927 07:03:06.142789 19964 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:34685
I20260927 07:03:06.142891 20093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:34685 every 8 connection(s)
I20260927 07:03:06.143206 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:06.144426 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.143956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:51733)
0927 07:03:06.144135 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.144229 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
W20260927 07:03:06.144652 20094 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37367 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:37367: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
I20260927 07:03:06.144971 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.144856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:50689)
0927 07:03:06.144902 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.144949 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 07:03:06.145287 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.145208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:39975)
0927 07:03:06.145250 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.145272 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:03:06.145416 20094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:03:06.146894 19986 log.cc:824] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:06.148838 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 19964
I20260927 07:03:06.148963 19986 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:03:06.149140 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:36737
--local_ip_for_outbound_sockets=127.3.57.3
--tserver_master_addrs=127.3.57.62:37367
--webserver_port=33281
--webserver_interface=127.3.57.3
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:06.149295 19986 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Bootstrap complete.
I20260927 07:03:06.150323 19986 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260927 07:03:06.152016 19986 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:06.152256 19986 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5b3510648a4f2daadba7f7190c4717, State: Initialized, Role: FOLLOWER
I20260927 07:03:06.152379 19986 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:06.152629 19986 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260927 07:03:06.152897 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.152800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:52493)
0927 07:03:06.152851 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.152875 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 07:03:06.153219 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.153140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:34097)
0927 07:03:06.153185 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.153206 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:03:06.229465 20098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:06.229663 20098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:06.229687 20098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:06.230788 20098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:06.230825 20098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:06.230850 20098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:06.232257 20098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:36737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=33281
--tserver_master_addrs=127.3.57.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:06.232443 20098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:06.232615 20098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:06.237675 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:06.237672 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:06.237689 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:06.237882 20098 server_base.cc:1034] running on GCE node
I20260927 07:03:06.238108 20098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:06.238387 20098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:06.239581 20098 hybrid_clock.cc:648] HybridClock initialized: now 1790492586239570 us; error 42 us; skew 500 ppm
Sep 27 07:03:06 dist-test-slave-mk3h krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:06.241976 20098 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:03:06.242404 20098 webserver.cc:460] Webserver started at http://127.3.57.3:33281/ using document root <none> and password file <none>
I20260927 07:03:06.242646 20098 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:06.242713 20098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:06.244122 20098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:03:06.244728 20114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:06.244900 20098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:06.245016 20098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "b1e427a27f554ca1b9c2ee45066cedf6"
format_stamp: "Formatted at 2026-09-27 07:02:24 on dist-test-slave-mk3h"
server_key: "8cdb4ee216752552549a6367f7429832"
server_key_iv: "663f972bdebbf92749d568574c6c015f"
server_key_version: "encryptionkey@0"
I20260927 07:03:06.245352 20098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:06.265380 20098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:06.267304 20098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:06.267472 20098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:06.267715 20098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:06.268029 20098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:06.268081 20098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:06.268123 20098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:06.268149 20098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:06.274010 20098 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:36737
I20260927 07:03:06.274055 20227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:36737 every 8 connection(s)
I20260927 07:03:06.274369 20098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:06.275380 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.275035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:42071)
0927 07:03:06.275182 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.275250 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
W20260927 07:03:06.275830 20228 heartbeater.cc:644] Failed to heartbeat to 127.3.57.62:37367 (0 consecutive failures): Network error: Failed to ping master at 127.3.57.62:37367: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
I20260927 07:03:06.276109 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.276019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:50815)
0927 07:03:06.276062 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.276089 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 07:03:06.276404 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:06.276319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:51161)
0927 07:03:06.276362 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:06.276385 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 07:03:06.276530 20228 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:03:06.284406 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20098
I20260927 07:03:07.085829 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.085604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:54051)
0927 07:03:07.085732 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.085767 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260927 07:03:07.140524 20233 raft_consensus.cc:482] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:07.140646 20233 raft_consensus.cc:504] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:07.140978 20233 leader_election.cc:288] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
Sep 27 07:03:07 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.3@KRBTEST.COM
Sep 27 07:03:07 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492585, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.2@KRBTEST.COM
I20260927 07:03:07.151674 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.141218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:59539 (local address 127.3.57.3:36737)
0927 07:03:07.141345 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:07.141357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:07.141382 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:03:07.141942 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:07.141944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:07.141991 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:07.142125 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:07.142761 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.144144 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.145212 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.145824 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.145945 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:07.148038 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:07.148042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:07.148043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:07.148056 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:07.149850 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150324 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:07.150326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:07.150327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:07.150406 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150673 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:07.150674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:07.150674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:07.150991 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:07.151076 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:07.151394 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 07:03:07.151547 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260927 07:03:07.151678 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.141105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36737 (local address 127.3.57.1:59539)
0927 07:03:07.141159 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.141171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:07.141225 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:07.142117 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:07.142117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:07.142383 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:07.142571 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:07.142577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.144242 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.144243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:07.145110 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:07.145113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.145907 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.145908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:07.145991 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:07.146483 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:07.146491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:07.147923 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:07.149938 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:07.149945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:07.149947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:07.150233 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:07.150470 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:07.150472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:07.150604 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:07.151080 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:07.151087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:07.151234 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 07:03:07.151346 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 07:03:07.151530 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":73}
I20260927 07:03:07.151826 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.141133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:34685 (local address 127.3.57.1:49077)
0927 07:03:07.141266 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.141279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:07.141335 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:07.141797 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:07.141799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:07.142361 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:07.142571 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:07.142577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.144283 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.144284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:07.145109 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:07.145112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.146045 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.146046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:07.146091 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:07.146630 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:07.146635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:07.148291 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:07.150528 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:07.150530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:07.150736 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:07.150959 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:07.150960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:07.151035 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:07.151452 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:07.151453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:07.151508 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:03:07.151613 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:03:07.151706 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":70,"thread_start_us":63,"threads_started":1}
I20260927 07:03:07.152194 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.141259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:49077 (local address 127.3.57.2:34685)
0927 07:03:07.141396 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:07.141404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:07.141421 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:03:07.141521 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:07.141524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:07.141589 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:07.141723 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:07.142701 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.144126 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.145261 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:07.145822 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:07.145939 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:07.148431 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:07.148439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:07.148440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:07.148451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:07.150426 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:07.150812 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:07.150814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:07.150814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:07.150887 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:07.151102 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:07.151102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:07.151103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:07.151381 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:07.151456 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:07.151670 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 07:03:07.152116 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260927 07:03:07.152554 20048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fa5b3510648a4f2daadba7f7190c4717" is_pre_election: true
I20260927 07:03:07.152709 20048 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 1.
W20260927 07:03:07.152813 19848 leader_election.cc:341] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737): Not found: Tablet not found: 97cda1bad9b74016a2693147dab8ee76
I20260927 07:03:07.152952 19847 leader_election.cc:302] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e0fa6b63a47145ea802b106ae315119e, fa5b3510648a4f2daadba7f7190c4717; no voters: b1e427a27f554ca1b9c2ee45066cedf6
I20260927 07:03:07.153086 20233 raft_consensus.cc:2781] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 07:03:07.153162 20233 raft_consensus.cc:482] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:07.153180 20233 raft_consensus.cc:3037] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:03:07.153716 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.153562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:50019)
0927 07:03:07.153662 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.153690 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260927 07:03:07.154448 20233 raft_consensus.cc:504] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:07.154594 20233 leader_election.cc:288] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 election: Requested vote from peers b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737), fa5b3510648a4f2daadba7f7190c4717 (127.3.57.2:34685)
I20260927 07:03:07.154778 20048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97cda1bad9b74016a2693147dab8ee76" candidate_uuid: "e0fa6b63a47145ea802b106ae315119e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fa5b3510648a4f2daadba7f7190c4717"
W20260927 07:03:07.154855 19848 leader_election.cc:341] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737): Not found: Tablet not found: 97cda1bad9b74016a2693147dab8ee76
I20260927 07:03:07.154862 20048 raft_consensus.cc:3037] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:03:07.155958 20048 raft_consensus.cc:2454] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0fa6b63a47145ea802b106ae315119e in term 2.
I20260927 07:03:07.156119 19847 leader_election.cc:302] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e0fa6b63a47145ea802b106ae315119e, fa5b3510648a4f2daadba7f7190c4717; no voters: b1e427a27f554ca1b9c2ee45066cedf6
I20260927 07:03:07.156250 20233 raft_consensus.cc:2781] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:03:07.156364 20233 raft_consensus.cc:686] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [term 2 LEADER]: Becoming Leader. State: Replica: e0fa6b63a47145ea802b106ae315119e, State: Running, Role: LEADER
I20260927 07:03:07.156479 20233 consensus_queue.cc:237] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:07.156664 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.156555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:48673)
0927 07:03:07.156611 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.156642 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:03:07.277310 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.276998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:49551)
0927 07:03:07.277207 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.277261 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260927 07:03:07.603681 20048 raft_consensus.cc:1261] T 97cda1bad9b74016a2693147dab8ee76 P fa5b3510648a4f2daadba7f7190c4717 [term 2 FOLLOWER]: Refusing update from remote peer e0fa6b63a47145ea802b106ae315119e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 07:03:07.604133 20233 consensus_queue.cc:1035] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 07:03:07.605096 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.604954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:53501)
0927 07:03:07.605036 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.605067 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 07:03:07.605933 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.605834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:51249)
0927 07:03:07.605877 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.605903 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 07:03:07.606062 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:07.605950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:41529)
0927 07:03:07.606004 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:07.606036 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 07:03:07.606292 19848 consensus_peers.cc:487] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e -> Peer b1e427a27f554ca1b9c2ee45066cedf6 (127.3.57.3:36737): Couldn't send request to peer b1e427a27f554ca1b9c2ee45066cedf6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97cda1bad9b74016a2693147dab8ee76. This is attempt 1: this message will repeat every 5th retry.
I20260927 07:03:08.046782 20245 ts_tablet_manager.cc:927] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Initiating tablet copy from peer e0fa6b63a47145ea802b106ae315119e (127.3.57.1:41789)
I20260927 07:03:08.047006 20245 tablet_copy_client.cc:316] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: tablet copy: Beginning tablet copy session from remote peer at address 127.3.57.1:41789
Sep 27 07:03:08 dist-test-slave-mk3h krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.1@KRBTEST.COM
I20260927 07:03:08.054862 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.047399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:41757 (local address 127.3.57.1:41789)
0927 07:03:08.047526 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:08.047532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:08.047580 (+ 48us) server_negotiation.cc:407] Connection header received
0927 07:03:08.047607 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:08.047609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:08.047624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:08.047685 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:08.048220 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:08.048932 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:08.049714 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:08.050038 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:08.050125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:08.052195 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:08.052202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:08.052202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:08.052209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:08.053578 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:08.053972 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:08.053975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:08.053975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:08.054038 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:08.054245 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:08.054246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:08.054246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:08.054395 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:08.054481 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:08.054683 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 07:03:08.054761 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":40,"threads_started":1}
I20260927 07:03:08.054875 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.047263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:41789 (local address 127.3.57.3:41757)
0927 07:03:08.047407 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.047421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:08.047519 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:08.047688 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:08.047692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:08.048075 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:08.048133 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:08.048136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:08.049041 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:08.049043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:08.049624 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:08.049626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:08.050122 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:08.050124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:08.050203 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:08.050669 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:08.050673 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:08.052097 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:08.053678 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:08.053685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:08.053687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:08.053885 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:08.054114 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:08.054115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:08.054115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:08.054176 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:08.054471 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:08.054478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:08.054529 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 07:03:08.054629 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:03:08.054761 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260927 07:03:08.055351 19933 tablet_copy_service.cc:140] P e0fa6b63a47145ea802b106ae315119e: Received BeginTabletCopySession request for tablet 97cda1bad9b74016a2693147dab8ee76 from peer b1e427a27f554ca1b9c2ee45066cedf6 ({username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:41757)
I20260927 07:03:08.055423 19933 tablet_copy_service.cc:161] P e0fa6b63a47145ea802b106ae315119e: Beginning new tablet copy session on tablet 97cda1bad9b74016a2693147dab8ee76 from peer b1e427a27f554ca1b9c2ee45066cedf6 at {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:41757: session id = b1e427a27f554ca1b9c2ee45066cedf6-97cda1bad9b74016a2693147dab8ee76
I20260927 07:03:08.055966 19933 tablet_copy_source_session.cc:205] T 97cda1bad9b74016a2693147dab8ee76 P e0fa6b63a47145ea802b106ae315119e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 07:03:08.056854 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97cda1bad9b74016a2693147dab8ee76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:08.059314 20245 tablet_copy_client.cc:791] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: tablet copy: Starting download of 0 data blocks...
I20260927 07:03:08.059487 20245 tablet_copy_client.cc:655] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: tablet copy: Starting download of 1 WAL segments...
I20260927 07:03:08.060308 20245 tablet_copy_client.cc:531] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 07:03:08.061015 20245 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Bootstrap starting.
I20260927 07:03:08.062413 20245 log.cc:824] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:08.064657 20245 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:03:08.064888 20245 tablet_bootstrap.cc:492] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Bootstrap complete.
I20260927 07:03:08.065023 20245 ts_tablet_manager.cc:1397] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 07:03:08.065953 20245 raft_consensus.cc:348] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:08.066087 20245 raft_consensus.cc:729] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e427a27f554ca1b9c2ee45066cedf6, State: Initialized, Role: FOLLOWER
I20260927 07:03:08.066231 20245 consensus_queue.cc:260] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1e427a27f554ca1b9c2ee45066cedf6" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36737 } } peers { permanent_uuid: "fa5b3510648a4f2daadba7f7190c4717" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 34685 } } peers { permanent_uuid: "e0fa6b63a47145ea802b106ae315119e" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 41789 } }
I20260927 07:03:08.066486 20245 ts_tablet_manager.cc:1428] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:08.066787 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.066680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:37649)
0927 07:03:08.066708 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.066753 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 07:03:08.066783 19933 tablet_copy_service.cc:342] P e0fa6b63a47145ea802b106ae315119e: Request end of tablet copy session b1e427a27f554ca1b9c2ee45066cedf6-97cda1bad9b74016a2693147dab8ee76 received from {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:41757
I20260927 07:03:08.066829 19933 tablet_copy_service.cc:433] P e0fa6b63a47145ea802b106ae315119e: ending tablet copy session b1e427a27f554ca1b9c2ee45066cedf6-97cda1bad9b74016a2693147dab8ee76 on tablet 97cda1bad9b74016a2693147dab8ee76 with peer b1e427a27f554ca1b9c2ee45066cedf6
I20260927 07:03:08.067159 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.067067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:55367)
0927 07:03:08.067101 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.067128 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 07:03:08.590322 20182 raft_consensus.cc:1203] T 97cda1bad9b74016a2693147dab8ee76 P b1e427a27f554ca1b9c2ee45066cedf6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 07:03:08.590374 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.590197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:33703)
0927 07:03:08.590296 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.590331 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260927 07:03:08.590924 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.590761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:35195)
0927 07:03:08.590857 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.590886 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260927 07:03:08.606619 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:08.606450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:37149)
0927 07:03:08.606545 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:08.606579 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
I20260927 07:03:09.590972 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:09.590770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:58567)
0927 07:03:09.590893 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:09.590925 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260927 07:03:09.591450 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:09.591265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:39209)
0927 07:03:09.591365 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:09.591413 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260927 07:03:09.607287 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:09.607070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:35619)
0927 07:03:09.607213 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:09.607248 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":50,"threads_started":1}
I20260927 07:03:10.591866 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:10.591658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:51287)
0927 07:03:10.591787 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:10.591811 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260927 07:03:10.592077 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:10.591880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:34989)
0927 07:03:10.592008 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:10.592046 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260927 07:03:10.607904 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:10.607718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:37271)
0927 07:03:10.607831 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:10.607865 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":46,"threads_started":1}
I20260927 07:03:11.592500 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:11.592289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:49453)
0927 07:03:11.592420 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:11.592454 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":52,"threads_started":1}
I20260927 07:03:11.592679 20264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:11.592483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:58763)
0927 07:03:11.592586 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:11.592622 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260927 07:03:11.592887 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:11.592796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:57693)
0927 07:03:11.592832 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:11.592861 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 07:03:11.608501 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:11.608313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:45859)
0927 07:03:11.608431 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:11.608462 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260927 07:03:12.593475 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:12.593249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:46895)
0927 07:03:12.593388 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:12.593430 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
I20260927 07:03:12.593498 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:12.593259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:57311)
0927 07:03:12.593425 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:12.593458 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260927 07:03:12.609089 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:12.608897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:55205)
0927 07:03:12.609016 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:12.609050 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260927 07:03:13.594120 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:13.593908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:43635)
0927 07:03:13.594034 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:13.594075 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260927 07:03:13.594144 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:13.593945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:33781)
0927 07:03:13.594074 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:13.594104 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 07:03:13.609743 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:13.609524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:53477)
0927 07:03:13.609649 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:13.609685 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260927 07:03:14.594880 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:14.594638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:45977)
0927 07:03:14.594789 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:14.594828 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260927 07:03:14.594916 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:14.594683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:40837)
0927 07:03:14.594836 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:14.594876 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260927 07:03:14.610384 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:14.610186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:33411)
0927 07:03:14.610304 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:14.610341 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
I20260927 07:03:15.595690 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:15.595426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:51325)
0927 07:03:15.595591 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:15.595636 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260927 07:03:15.595719 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:15.595513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:44451)
0927 07:03:15.595644 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:15.595679 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260927 07:03:15.611007 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:15.610829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:57023)
0927 07:03:15.610931 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:15.610967 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260927 07:03:16.596393 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:16.596197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:33185)
0927 07:03:16.596304 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:16.596348 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":41,"threads_started":1}
I20260927 07:03:16.596400 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:16.596167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:51907)
0927 07:03:16.596301 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:16.596359 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260927 07:03:16.611682 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:16.611438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:52945)
0927 07:03:16.611596 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:16.611634 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":52,"threads_started":1}
I20260927 07:03:17.597041 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:17.596847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:45705)
0927 07:03:17.596960 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:17.596997 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260927 07:03:17.597061 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:17.596830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:54943)
0927 07:03:17.596992 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:17.597021 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260927 07:03:17.612274 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:17.612075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:49097)
0927 07:03:17.612199 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:17.612232 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260927 07:03:18.597632 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:18.597425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:46641)
0927 07:03:18.597557 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:18.597587 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260927 07:03:18.597837 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:18.597594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:55703)
0927 07:03:18.597731 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:18.597791 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260927 07:03:18.612895 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:18.612703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:43947)
0927 07:03:18.612815 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:18.612850 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260927 07:03:19.598395 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:19.598121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:49373)
0927 07:03:19.598302 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:19.598339 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260927 07:03:19.598553 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:19.598320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:59103)
0927 07:03:19.598453 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:19.598483 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260927 07:03:19.613497 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:19.613312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:60815)
0927 07:03:19.613419 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:19.613454 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260927 07:03:20.599170 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:20.598963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.3:56775)
0927 07:03:20.599087 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:20.599121 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260927 07:03:20.599187 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:20.598949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.1:39429)
0927 07:03:20.599118 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:20.599147 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260927 07:03:20.614089 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:20.613908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37367 (local address 127.3.57.2:40701)
0927 07:03:20.614016 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:20.614043 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.62:37367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20260927 07:03:21.294982 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19795
I20260927 07:03:21.299803 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 19964
I20260927 07:03:21.304411 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20098
2026-09-27T07:03:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58355 ms)
[----------] 30 tests from SecurityITest (259339 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:21 dist-test-slave-mk3h krb5kdc[20297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:21 dist-test-slave-mk3h krb5kdc[20297](info): set up 2 sockets
Sep 27 07:03:21 dist-test-slave-mk3h krb5kdc[20297](info): commencing operation
krb5kdc: starting...
W20260927 07:03:22.331014 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:22.442505 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35885
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:22.519520 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:22.519769 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:22.519804 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:22.520884 20310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:22.520954 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:22.520978 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:22.520998 20310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:22.521023 20310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:22.522409 20310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:35885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:22.522675 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:22.522955 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:22.527441 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:22.527612 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:22.527596 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:22.527760 20310 server_base.cc:1034] running on GCE node
I20260927 07:03:22.527969 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:22.528244 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:22.529395 20310 hybrid_clock.cc:648] HybridClock initialized: now 1790492602529374 us; error 41 us; skew 500 ppm
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:22.538653 20310 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:22.539060 20310 webserver.cc:460] Webserver started at http://127.3.57.62:42271/ using document root <none> and password file <none>
I20260927 07:03:22.539323 20310 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:22.539393 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:22.539600 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:22.541006 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "2040afccac174254b7b6cd783bb65cf0"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "98d1294cae85310f8fcd6c72041346d1"
server_key_iv: "347838ed406a2f3a16ed03f1835be2d5"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.541415 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "2040afccac174254b7b6cd783bb65cf0"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "98d1294cae85310f8fcd6c72041346d1"
server_key_iv: "347838ed406a2f3a16ed03f1835be2d5"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.543040 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:22.543809 20325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.543992 20310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:22.544086 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "2040afccac174254b7b6cd783bb65cf0"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "98d1294cae85310f8fcd6c72041346d1"
server_key_iv: "347838ed406a2f3a16ed03f1835be2d5"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.544173 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:22.557832 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:22.559494 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:22.559660 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:22.563436 20310 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:35885
I20260927 07:03:22.563467 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:35885 every 8 connection(s)
I20260927 07:03:22.563836 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:22.564630 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:22.566700 20378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: Bootstrap starting.
I20260927 07:03:22.566830 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20310
I20260927 07:03:22.566954 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:22.567163 3300 external_mini_cluster.cc:1428] Setting key b2fb036684af1b25a5e746582e396cfb
I20260927 07:03:22.567375 20378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:22.567957 20378 log.cc:824] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:22.568661 20378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: No bootstrap required, opened a new log
I20260927 07:03:22.570892 20378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } }
I20260927 07:03:22.571007 20378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:22.571028 20378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2040afccac174254b7b6cd783bb65cf0, State: Initialized, Role: FOLLOWER
I20260927 07:03:22.571156 20378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } }
I20260927 07:03:22.571228 20378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:22.571269 20378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:22.571312 20378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:22.571931 20378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } }
I20260927 07:03:22.572036 20378 leader_election.cc:302] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2040afccac174254b7b6cd783bb65cf0; no voters:
I20260927 07:03:22.572180 20378 leader_election.cc:288] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:22.572245 20383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:22.572405 20383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 2040afccac174254b7b6cd783bb65cf0, State: Running, Role: LEADER
I20260927 07:03:22.572415 20378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:22.572532 20383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } }
I20260927 07:03:22.573279 20385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2040afccac174254b7b6cd783bb65cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } } }
I20260927 07:03:22.573294 20384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2040afccac174254b7b6cd783bb65cf0. Latest consensus state: current_term: 1 leader_uuid: "2040afccac174254b7b6cd783bb65cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2040afccac174254b7b6cd783bb65cf0" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 35885 } } }
I20260927 07:03:22.573375 20385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:22.573405 20384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:22.573586 20389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:22.574079 20389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:22.575482 20389 catalog_manager.cc:1347] Generated new cluster ID: f6f500872ad64365bc0597f62f088f6c
I20260927 07:03:22.575534 20389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:22.580184 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:22.567668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.3.57.62:35885)
0927 07:03:22.567837 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:22.567845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:22.567887 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:03:22.568439 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:22.568447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:22.568506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:22.568648 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:22.569255 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.570682 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.571651 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.572262 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.572694 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:22.575175 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:22.575179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:22.575179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:22.575190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:22.578263 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.578644 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.578647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.578647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.578738 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.578949 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.578950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.578950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.579216 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:22.579777 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:22.579865 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 07:03:22.579988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260927 07:03:22.596020 20389 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:22.596490 20389 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:22.603039 20389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: Generated new TSK 0
I20260927 07:03:22.603217 20389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:03:22.630716 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:22.708212 20404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:22.708388 20404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:22.708410 20404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:22.709467 20404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:22.709501 20404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:22.709527 20404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:22.710853 20404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:22.711031 20404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:22.711205 20404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:22.715869 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:22.715955 20404 server_base.cc:1034] running on GCE node
W20260927 07:03:22.715965 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:22.716109 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:22.716339 20404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:22.716570 20404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:22.717710 20404 hybrid_clock.cc:648] HybridClock initialized: now 1790492602717685 us; error 37 us; skew 500 ppm
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:22.720041 20404 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:03:22.720372 20404 webserver.cc:460] Webserver started at http://127.3.57.1:46129/ using document root <none> and password file <none>
I20260927 07:03:22.720551 20404 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:22.720588 20404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:22.720669 20404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:22.721549 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "df22c3ed84d844d2b32319abb70102dd"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "fe2c151585a2eed5a4e96c8d7fd99835"
server_key_iv: "9d4c2f3a0830208f70ea9288fd50c855"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.721825 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "df22c3ed84d844d2b32319abb70102dd"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "fe2c151585a2eed5a4e96c8d7fd99835"
server_key_iv: "9d4c2f3a0830208f70ea9288fd50c855"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.722977 20404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:22.723762 20419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.723903 20404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:22.724002 20404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "df22c3ed84d844d2b32319abb70102dd"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "fe2c151585a2eed5a4e96c8d7fd99835"
server_key_iv: "9d4c2f3a0830208f70ea9288fd50c855"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.724078 20404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:22.753429 20404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:22.755129 20404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:22.755234 20404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:22.755398 20404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:22.755733 20404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:22.755764 20404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.755784 20404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:22.755831 20404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.761345 20404 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:34521
I20260927 07:03:22.761445 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:34521 every 8 connection(s)
I20260927 07:03:22.761720 20404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:22.764927 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20404
I20260927 07:03:22.765035 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:22.765251 3300 external_mini_cluster.cc:1428] Setting key d4063f3faf88c4ff8ec346a755f3b21f
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:22.771379 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:22.762398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35885 (local address 127.3.57.1:40665)
0927 07:03:22.762568 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:22.762600 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:22.763027 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:22.763207 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:22.763210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:22.763738 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:22.763908 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:22.763912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.764584 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.764585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:22.765748 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:22.765751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.766167 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.766168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:22.766332 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:22.766786 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:22.766795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:22.768330 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:22.770135 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:22.770141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:22.770142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:22.770343 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:22.770566 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:22.770566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:22.770566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:22.770656 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:22.770947 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:22.770951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:22.771057 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 07:03:22.771131 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:03:22.771237 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260927 07:03:22.771420 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:22.762447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:40665 (local address 127.3.57.62:35885)
0927 07:03:22.762520 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:22.762522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:22.762677 (+ 155us) server_negotiation.cc:407] Connection header received
0927 07:03:22.763101 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:22.763102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:22.763111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:22.763159 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:22.763977 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.764508 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.765828 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.766082 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.766135 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:22.768430 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:22.768432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:22.768433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:22.768440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:22.770043 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.770426 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.770428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.770428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.770493 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.770724 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.770725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.770726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.770872 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:22.770946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:22.771182 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 07:03:22.771271 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:03:22.772161 20533 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35885
I20260927 07:03:22.772244 20533 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:22.772406 20533 heartbeater.cc:511] Master 127.3.57.62:35885 requested a full tablet report, sending...
I20260927 07:03:22.772897 20342 ts_manager.cc:194] Registered new tserver with Master: df22c3ed84d844d2b32319abb70102dd (127.3.57.1:34521)
I20260927 07:03:22.773525 20342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:40665
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:03:22.788643 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:22.866712 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:22.866936 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:22.866976 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:22.868130 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:22.868208 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:22.868252 20538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:22.869710 20538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:22.869962 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:22.870172 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:22.875156 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:22.875219 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:22.875347 20538 server_base.cc:1034] running on GCE node
W20260927 07:03:22.875492 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:22.875726 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:22.875953 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:22.877136 20538 hybrid_clock.cc:648] HybridClock initialized: now 1790492602877111 us; error 40 us; skew 500 ppm
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:22.879483 20538 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:03:22.879953 20538 webserver.cc:460] Webserver started at http://127.3.57.2:42911/ using document root <none> and password file <none>
I20260927 07:03:22.880174 20538 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:22.880218 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:22.880368 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:22.881335 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "6ccfb845ecd24c65a2b5e2748b288c8a"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "0af018603c006493af63faa21bfc5613"
server_key_iv: "9568ceba3fdec5ff3ee2bd2fbd9c0a71"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.881672 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "6ccfb845ecd24c65a2b5e2748b288c8a"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "0af018603c006493af63faa21bfc5613"
server_key_iv: "9568ceba3fdec5ff3ee2bd2fbd9c0a71"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.882895 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:22.883572 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.883747 20538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:22.883812 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "6ccfb845ecd24c65a2b5e2748b288c8a"
format_stamp: "Formatted at 2026-09-27 07:03:22 on dist-test-slave-mk3h"
server_key: "0af018603c006493af63faa21bfc5613"
server_key_iv: "9568ceba3fdec5ff3ee2bd2fbd9c0a71"
server_key_version: "encryptionkey@0"
I20260927 07:03:22.883898 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:22.899634 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:22.901319 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:22.901464 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:22.901685 20538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:22.901978 20538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:22.902015 20538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.902034 20538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:22.902046 20538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:22.907611 20538 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:33101
I20260927 07:03:22.907661 20666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:33101 every 8 connection(s)
I20260927 07:03:22.907953 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:22.912899 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20538
I20260927 07:03:22.912978 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:22.913185 3300 external_mini_cluster.cc:1428] Setting key 20da324a162a4eb98549d08831d67c39
Sep 27 07:03:22 dist-test-slave-mk3h krb5kdc[20297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:22.917654 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:22.908622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:53227 (local address 127.3.57.62:35885)
0927 07:03:22.908663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:22.908664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:22.908826 (+ 162us) server_negotiation.cc:407] Connection header received
0927 07:03:22.909269 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:22.909270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:22.909278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:22.909342 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:22.910621 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.911124 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.912071 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.912271 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.912322 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:22.914609 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:22.914610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:22.914611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:22.914620 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:22.916221 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.916623 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.916625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.916625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.916687 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:22.916980 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:22.916981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:22.916981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:22.917128 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:22.917198 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:22.917429 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 07:03:22.917519 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:22.917797 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:22.908595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35885 (local address 127.3.57.2:53227)
0927 07:03:22.908728 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:22.908764 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:22.909193 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:22.909400 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:22.909403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:22.910160 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:22.910473 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:22.910479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.911195 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.911196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:22.912004 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:22.912006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:22.912339 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:22.912339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:22.912459 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:22.912893 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:22.912900 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:22.914520 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:22.916323 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:22.916329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:22.916331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:22.916547 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:22.916752 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:22.916753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:22.916753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:22.916840 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:22.917193 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:22.917197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:22.917298 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 07:03:22.917374 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:03:22.917517 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260927 07:03:22.918278 20667 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35885
I20260927 07:03:22.918388 20667 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:22.918568 20667 heartbeater.cc:511] Master 127.3.57.62:35885 requested a full tablet report, sending...
I20260927 07:03:22.918964 20342 ts_manager.cc:194] Registered new tserver with Master: 6ccfb845ecd24c65a2b5e2748b288c8a (127.3.57.2:33101)
I20260927 07:03:22.919332 20342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:53227
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:03:22.938076 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:23.015196 20672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:23.015408 20672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:23.015443 20672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:23.016551 20672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:23.016619 20672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:23.016664 20672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:23.018096 20672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:23.018357 20672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:23.018573 20672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:23.023367 20680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.023365 20677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.023645 20672 server_base.cc:1034] running on GCE node
W20260927 07:03:23.023365 20678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.023983 20672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:23.024257 20672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:23.025418 20672 hybrid_clock.cc:648] HybridClock initialized: now 1790492603025406 us; error 30 us; skew 500 ppm
Sep 27 07:03:23 dist-test-slave-mk3h krb5kdc[20297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:23.028023 20672 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:03:23.028446 20672 webserver.cc:460] Webserver started at http://127.3.57.3:45877/ using document root <none> and password file <none>
I20260927 07:03:23.028640 20672 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:23.028680 20672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:23.028765 20672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:23.029680 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "865da1205c0342ef8c238ee66a77051d"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "cf83893708a73f64d4b406b923e16627"
server_key_iv: "5fc608d2a9deb73d00423b59223426be"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.029953 20672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "865da1205c0342ef8c238ee66a77051d"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "cf83893708a73f64d4b406b923e16627"
server_key_iv: "5fc608d2a9deb73d00423b59223426be"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.031092 20672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:23.031776 20687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.031951 20672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:23.032039 20672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "865da1205c0342ef8c238ee66a77051d"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "cf83893708a73f64d4b406b923e16627"
server_key_iv: "5fc608d2a9deb73d00423b59223426be"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.032091 20672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:23.063455 20672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:23.065313 20672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:23.065433 20672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:23.065609 20672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:23.065920 20672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:23.065953 20672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.065975 20672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:23.066027 20672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.071404 20672 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:35645
I20260927 07:03:23.071530 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:35645 every 8 connection(s)
I20260927 07:03:23.071815 20672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:23.072220 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20672
I20260927 07:03:23.072328 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:23.072511 3300 external_mini_cluster.cc:1428] Setting key e5a9a31d228d154efe9e2c9309cb4c0d
Sep 27 07:03:23 dist-test-slave-mk3h krb5kdc[20297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492603, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:23.081161 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.072591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:44815 (local address 127.3.57.62:35885)
0927 07:03:23.072646 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.072648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.073028 (+ 380us) server_negotiation.cc:407] Connection header received
0927 07:03:23.073596 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.073598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.073610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.073675 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.074638 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.075137 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.076172 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.076356 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.076412 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.078420 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.078421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.078421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:23.078430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.079770 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:23.080213 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:23.080221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:23.080221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:23.080272 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:23.080519 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:23.080519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:23.080520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:23.080631 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.080702 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.080979 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.081070 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 07:03:23.081316 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.072564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:35885 (local address 127.3.57.3:44815)
0927 07:03:23.072792 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.072840 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.073458 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.073799 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.073804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.074376 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:23.074553 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.074557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.075228 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.075228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.076103 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.076105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.076427 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.076428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.076527 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.076985 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:23.076994 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:23.078321 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:23.079879 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:23.079885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:23.079887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:23.080101 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:23.080339 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:23.080339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:23.080340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:23.080443 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:23.080704 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:23.080711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:23.080842 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.080940 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.081092 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260927 07:03:23.081765 20801 heartbeater.cc:347] Connected to a master server at 127.3.57.62:35885
I20260927 07:03:23.081861 20801 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:23.082034 20801 heartbeater.cc:511] Master 127.3.57.62:35885 requested a full tablet report, sending...
I20260927 07:03:23.082438 20342 ts_manager.cc:194] Registered new tserver with Master: 865da1205c0342ef8c238ee66a77051d (127.3.57.3:35645)
I20260927 07:03:23.082823 20342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:44815
I20260927 07:03:23.084471 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:23.091617 20381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.085761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.3.57.62:35885)
0927 07:03:23.085826 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.085828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.085931 (+ 103us) server_negotiation.cc:407] Connection header received
0927 07:03:23.086020 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.086021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.086030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.086093 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.086746 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.087219 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.088096 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.088293 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.088374 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.089414 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.089415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.089416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:23.089423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.090485 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:23.090814 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:23.090815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:23.090815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:23.090874 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:23.091070 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:23.091070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:23.091071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:23.091213 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.091332 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.091417 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.091493 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:03:23.092993 20342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:23.094460 20342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:23.100849 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.097030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.3.57.3:35645)
0927 07:03:23.097403 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.097407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.097418 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:03:23.097486 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.097488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.097501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.097573 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.097727 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.098488 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.099846 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.100474 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.100611 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.100736 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.100773 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":99,"threads_started":1}
I20260927 07:03:23.100845 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.096919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:35645 (local address 127.0.0.1:57554)
0927 07:03:23.097367 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.097378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.097422 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.097589 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.097591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.097593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.097653 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.097655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.098585 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.098586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.099752 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.099755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.100556 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.100633 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.100645 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.100713 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.100766 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":52,"threads_started":1}
I20260927 07:03:23.101189 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.096780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:33101 (local address 127.0.0.1:49558)
0927 07:03:23.097152 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.097198 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.097258 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.097590 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.097593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.097595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.097658 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.097660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.098964 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.098965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.100005 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.100007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.100954 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.101030 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.101034 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.101080 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.101118 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":6,"thread_start_us":41,"threads_started":1}
I20260927 07:03:23.101491 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.097190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.3.57.2:33101)
0927 07:03:23.097356 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.097361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.097370 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:23.097410 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.097412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.097428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.097500 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.097744 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.098860 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100086 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.100790 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100861 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.100913 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.101345 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.101389 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260927 07:03:23.101557 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.096568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:34521 (local address 127.0.0.1:39226)
0927 07:03:23.096939 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.097214 (+ 275us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.097273 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.097813 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.097813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.097815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.097858 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.097859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.099061 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.099062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.100144 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.100146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.101346 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.101347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.101412 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.101416 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.101445 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.101480 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":66,"threads_started":1}
I20260927 07:03:23.101656 20735 tablet_service.cc:1467] Processing CreateTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 (DEFAULT_TABLE table=test-table [id=e9a95a575ae7424ebfb1739cb000baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.101927 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.097107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.3.57.1:34521)
0927 07:03:23.097660 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.097665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.097674 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:23.097711 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.097713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.097729 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.097815 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.097905 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.098966 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.100230 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.101247 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.101703 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.101778 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.101803 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.101846 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":84,"threads_started":1}
I20260927 07:03:23.102010 20735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f9a8e8507e4c41b150e2db45f40e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.102077 20601 tablet_service.cc:1467] Processing CreateTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 (DEFAULT_TABLE table=test-table [id=e9a95a575ae7424ebfb1739cb000baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.102393 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f9a8e8507e4c41b150e2db45f40e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.102520 20467 tablet_service.cc:1467] Processing CreateTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 (DEFAULT_TABLE table=test-table [id=e9a95a575ae7424ebfb1739cb000baf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.102809 20467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f9a8e8507e4c41b150e2db45f40e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.105103 20815 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Bootstrap starting.
I20260927 07:03:23.105434 20816 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Bootstrap starting.
I20260927 07:03:23.105827 20815 tablet_bootstrap.cc:654] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.105998 20817 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: Bootstrap starting.
I20260927 07:03:23.106083 20816 tablet_bootstrap.cc:654] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.106621 20817 tablet_bootstrap.cc:654] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.106698 20816 log.cc:824] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.106748 20815 log.cc:824] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.107187 20817 log.cc:824] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.107436 20815 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: No bootstrap required, opened a new log
I20260927 07:03:23.107522 20815 ts_tablet_manager.cc:1397] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:23.107918 20817 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: No bootstrap required, opened a new log
I20260927 07:03:23.107926 20816 tablet_bootstrap.cc:492] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: No bootstrap required, opened a new log
I20260927 07:03:23.107987 20817 ts_tablet_manager.cc:1397] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:23.107987 20816 ts_tablet_manager.cc:1397] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:03:23.108942 20815 raft_consensus.cc:348] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.109105 20815 raft_consensus.cc:374] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.109150 20815 raft_consensus.cc:729] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ccfb845ecd24c65a2b5e2748b288c8a, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.109258 20815 consensus_queue.cc:260] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.109544 20815 ts_tablet_manager.cc:1428] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:23.109676 20667 heartbeater.cc:503] Master 127.3.57.62:35885 was elected leader, sending a full tablet report...
I20260927 07:03:23.109855 20817 raft_consensus.cc:348] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.109874 20816 raft_consensus.cc:348] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.110021 20816 raft_consensus.cc:374] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.110021 20817 raft_consensus.cc:374] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.110056 20816 raft_consensus.cc:729] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865da1205c0342ef8c238ee66a77051d, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.110056 20817 raft_consensus.cc:729] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df22c3ed84d844d2b32319abb70102dd, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.110160 20816 consensus_queue.cc:260] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.110180 20817 consensus_queue.cc:260] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.110425 20817 ts_tablet_manager.cc:1428] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:03:23.110426 20816 ts_tablet_manager.cc:1428] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:03:23.110584 20533 heartbeater.cc:503] Master 127.3.57.62:35885 was elected leader, sending a full tablet report...
I20260927 07:03:23.110586 20801 heartbeater.cc:503] Master 127.3.57.62:35885 was elected leader, sending a full tablet report...
W20260927 07:03:23.158546 20668 tablet.cc:2369] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:03:23.218771 20823 raft_consensus.cc:482] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:23.218907 20823 raft_consensus.cc:504] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.219349 20823 leader_election.cc:288] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ccfb845ecd24c65a2b5e2748b288c8a (127.3.57.2:33101), 865da1205c0342ef8c238ee66a77051d (127.3.57.3:35645)
I20260927 07:03:23.222282 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.219610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:33101 (local address 127.3.57.1:60299)
0927 07:03:23.219679 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.219694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.219752 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.219924 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.219925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.219926 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.220020 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.220022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.220693 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.220693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.221436 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.221438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.222007 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.222008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.222088 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.222093 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.222171 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.222213 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 07:03:23.222282 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.219713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:60299 (local address 127.3.57.2:33101)
0927 07:03:23.219762 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.219764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.219774 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:23.219817 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.219818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.219831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.219878 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.220088 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.220598 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.221536 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.221912 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.221998 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.222008 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.222175 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.222213 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:03:23.222473 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.219826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:32983 (local address 127.3.57.3:35645)
0927 07:03:23.219879 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.219882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.220032 (+ 150us) server_negotiation.cc:407] Connection header received
0927 07:03:23.220102 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.220103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.220114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.220145 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.220373 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.220816 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.221740 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.222113 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.222208 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.222217 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.222371 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.222410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:03:23.222482 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.219751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:35645 (local address 127.3.57.1:32983)
0927 07:03:23.219938 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.219950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.220021 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.220219 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.220220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.220221 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.220293 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.220294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.220908 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.220909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.221645 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.221647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.222211 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.222212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.222291 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.222294 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.222363 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.222402 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":128,"threads_started":1}
I20260927 07:03:23.222630 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f9a8e8507e4c41b150e2db45f40e48" candidate_uuid: "df22c3ed84d844d2b32319abb70102dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" is_pre_election: true
I20260927 07:03:23.222806 20621 raft_consensus.cc:2454] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df22c3ed84d844d2b32319abb70102dd in term 0.
I20260927 07:03:23.223104 20422 leader_election.cc:302] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ccfb845ecd24c65a2b5e2748b288c8a, df22c3ed84d844d2b32319abb70102dd; no voters:
I20260927 07:03:23.223201 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f9a8e8507e4c41b150e2db45f40e48" candidate_uuid: "df22c3ed84d844d2b32319abb70102dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865da1205c0342ef8c238ee66a77051d" is_pre_election: true
I20260927 07:03:23.223253 20823 raft_consensus.cc:2781] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:23.223315 20823 raft_consensus.cc:482] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:23.223338 20823 raft_consensus.cc:3037] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.223395 20755 raft_consensus.cc:2454] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df22c3ed84d844d2b32319abb70102dd in term 0.
I20260927 07:03:23.224190 20823 raft_consensus.cc:504] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.224370 20823 leader_election.cc:288] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [CANDIDATE]: Term 1 election: Requested vote from peers 6ccfb845ecd24c65a2b5e2748b288c8a (127.3.57.2:33101), 865da1205c0342ef8c238ee66a77051d (127.3.57.3:35645)
I20260927 07:03:23.224567 20755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f9a8e8507e4c41b150e2db45f40e48" candidate_uuid: "df22c3ed84d844d2b32319abb70102dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865da1205c0342ef8c238ee66a77051d"
I20260927 07:03:23.224548 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f9a8e8507e4c41b150e2db45f40e48" candidate_uuid: "df22c3ed84d844d2b32319abb70102dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a"
I20260927 07:03:23.224633 20755 raft_consensus.cc:3037] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.224653 20621 raft_consensus.cc:3037] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.225319 20755 raft_consensus.cc:2454] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df22c3ed84d844d2b32319abb70102dd in term 1.
I20260927 07:03:23.225438 20621 raft_consensus.cc:2454] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df22c3ed84d844d2b32319abb70102dd in term 1.
I20260927 07:03:23.225486 20422 leader_election.cc:302] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865da1205c0342ef8c238ee66a77051d, df22c3ed84d844d2b32319abb70102dd; no voters:
I20260927 07:03:23.225600 20823 raft_consensus.cc:2781] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:23.225728 20823 raft_consensus.cc:686] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 1 LEADER]: Becoming Leader. State: Replica: df22c3ed84d844d2b32319abb70102dd, State: Running, Role: LEADER
I20260927 07:03:23.225826 20823 consensus_queue.cc:237] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } }
I20260927 07:03:23.226711 20342 catalog_manager.cc:5495] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd reported cstate change: term changed from 0 to 1, leader changed from <none> to df22c3ed84d844d2b32319abb70102dd (127.3.57.1). New cstate: current_term: 1 leader_uuid: "df22c3ed84d844d2b32319abb70102dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df22c3ed84d844d2b32319abb70102dd" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260927 07:03:23.247483 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.244896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.3.57.1:34521)
0927 07:03:23.244961 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.244964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.244974 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:23.245050 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.245055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.245074 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.245149 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:23.245292 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.245760 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.246612 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.246830 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.246916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.247101 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:23.247201 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:23.247294 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.247337 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.247357 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:03:23.249414 20621 raft_consensus.cc:1261] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 1 FOLLOWER]: Refusing update from remote peer df22c3ed84d844d2b32319abb70102dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:23.249769 20755 raft_consensus.cc:1261] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 1 FOLLOWER]: Refusing update from remote peer df22c3ed84d844d2b32319abb70102dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:23.249781 20823 consensus_queue.cc:1035] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ccfb845ecd24c65a2b5e2748b288c8a" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:23.250115 20823 consensus_queue.cc:1035] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "865da1205c0342ef8c238ee66a77051d" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:23.252254 20828 mvcc.cc:205] Tried to move back new op lower bound from 7333857702906572800 to 7333857702813585408. Current Snapshot: MvccSnapshot[applied={T|T < 7333857702906572800}]
I20260927 07:03:23.252930 20830 mvcc.cc:205] Tried to move back new op lower bound from 7333857702906572800 to 7333857702813585408. Current Snapshot: MvccSnapshot[applied={T|T < 7333857702906572800}]
I20260927 07:03:23.254719 20341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:23.254822 20341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:23.255626 20341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2040afccac174254b7b6cd783bb65cf0: Sending DeleteTablet for 3 replicas of tablet e3f9a8e8507e4c41b150e2db45f40e48
I20260927 07:03:23.255926 20467 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:39226
I20260927 07:03:23.256014 20601 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:49558
I20260927 07:03:23.256065 20735 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f9a8e8507e4c41b150e2db45f40e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57554
I20260927 07:03:23.256196 20837 tablet_replica.cc:330] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd: stopping tablet replica
I20260927 07:03:23.256273 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20404
I20260927 07:03:23.256280 20838 tablet_replica.cc:330] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: stopping tablet replica
I20260927 07:03:23.256296 20837 raft_consensus.cc:2229] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:03:23.256376 20838 raft_consensus.cc:2229] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:23.256385 20837 raft_consensus.cc:2258] T e3f9a8e8507e4c41b150e2db45f40e48 P df22c3ed84d844d2b32319abb70102dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:23.256477 20838 raft_consensus.cc:2258] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:23.256675 20839 tablet_replica.cc:330] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: stopping tablet replica
I20260927 07:03:23.256753 20839 raft_consensus.cc:2229] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:23.256847 20839 raft_consensus.cc:2258] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:23.257149 20838 ts_tablet_manager.cc:1905] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:23.258776 20839 ts_tablet_manager.cc:1905] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:23.259140 20838 ts_tablet_manager.cc:1918] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:23.259220 20838 log.cc:1196] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/e3f9a8e8507e4c41b150e2db45f40e48
I20260927 07:03:23.259500 20838 ts_tablet_manager.cc:1939] T e3f9a8e8507e4c41b150e2db45f40e48 P 6ccfb845ecd24c65a2b5e2748b288c8a: Deleting consensus metadata
I20260927 07:03:23.260012 20328 catalog_manager.cc:4841] TS 6ccfb845ecd24c65a2b5e2748b288c8a (127.3.57.2:33101): tablet e3f9a8e8507e4c41b150e2db45f40e48 (table test-table [id=e9a95a575ae7424ebfb1739cb000baf5]) successfully deleted
I20260927 07:03:23.260588 20839 ts_tablet_manager.cc:1918] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:23.260674 20839 log.cc:1196] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/e3f9a8e8507e4c41b150e2db45f40e48
I20260927 07:03:23.260910 20839 ts_tablet_manager.cc:1939] T e3f9a8e8507e4c41b150e2db45f40e48 P 865da1205c0342ef8c238ee66a77051d: Deleting consensus metadata
I20260927 07:03:23.261281 20328 catalog_manager.cc:4841] TS 865da1205c0342ef8c238ee66a77051d (127.3.57.3:35645): tablet e3f9a8e8507e4c41b150e2db45f40e48 (table test-table [id=e9a95a575ae7424ebfb1739cb000baf5]) successfully deleted
W20260927 07:03:23.263602 20328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:34521 (error 108)
I20260927 07:03:23.263911 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20538
I20260927 07:03:23.264082 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.263935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:34521 (local address 127.0.0.1:39236)
0927 07:03:23.263998 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.264042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 07:03:23.264582 20328 catalog_manager.cc:4584] TS df22c3ed84d844d2b32319abb70102dd (127.3.57.1:34521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3f9a8e8507e4c41b150e2db45f40e48: Network error: Client connection negotiation failed: client connection to 127.3.57.1:34521: connect: Connection refused (error 111)
I20260927 07:03:23.269366 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20672
I20260927 07:03:23.273893 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20310
2026-09-27T07:03:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1969 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T07:03:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:23Z Disabled control of system clock
I20260927 07:03:23.302173 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37025
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:23.382928 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:23.383140 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:23.383162 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:23.384258 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:23.384291 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:23.384305 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:23.384316 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:23.384328 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:23.385563 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:37025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:23.385742 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:23.386034 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:23.390815 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.390815 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.390810 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.391563 20844 server_base.cc:1034] running on GCE node
I20260927 07:03:23.391801 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:23.392059 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:23.393205 20844 hybrid_clock.cc:648] HybridClock initialized: now 1790492603393184 us; error 36 us; skew 500 ppm
I20260927 07:03:23.394532 20844 webserver.cc:460] Webserver started at http://127.3.57.62:33915/ using document root <none> and password file <none>
I20260927 07:03:23.394793 20844 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:23.394845 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:23.395012 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:23.396232 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "91094b23ca054b2c8a2f14e1bb117621"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "fa726f0c62d803fc952e94f752386b3c"
server_key_iv: "4fe498425024dc80ae1f1502723e023d"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.396622 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "91094b23ca054b2c8a2f14e1bb117621"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "fa726f0c62d803fc952e94f752386b3c"
server_key_iv: "4fe498425024dc80ae1f1502723e023d"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.397946 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:23.398725 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.398900 20844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:23.399012 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "91094b23ca054b2c8a2f14e1bb117621"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "fa726f0c62d803fc952e94f752386b3c"
server_key_iv: "4fe498425024dc80ae1f1502723e023d"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.399091 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:23.416334 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:23.416674 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:23.416834 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:23.420975 20844 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:37025
I20260927 07:03:23.421033 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:37025 every 8 connection(s)
I20260927 07:03:23.421425 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:23.422399 20911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.424695 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: Bootstrap starting.
I20260927 07:03:23.425233 20911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.425474 20911 log.cc:824] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.426100 20911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: No bootstrap required, opened a new log
I20260927 07:03:23.426460 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20844
I20260927 07:03:23.426590 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:23.426805 3300 external_mini_cluster.cc:1428] Setting key d058452648f229d6bf04bedd78124116
I20260927 07:03:23.428305 20911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } }
I20260927 07:03:23.428501 20911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.428546 20911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91094b23ca054b2c8a2f14e1bb117621, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.428670 20911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } }
I20260927 07:03:23.428752 20911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:23.428786 20911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:23.428836 20911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.429563 20911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } }
I20260927 07:03:23.429664 20911 leader_election.cc:302] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91094b23ca054b2c8a2f14e1bb117621; no voters:
I20260927 07:03:23.429817 20911 leader_election.cc:288] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:23.429874 20916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:23.430044 20911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:23.430066 20916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [term 1 LEADER]: Becoming Leader. State: Replica: 91094b23ca054b2c8a2f14e1bb117621, State: Running, Role: LEADER
I20260927 07:03:23.430195 20916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } }
I20260927 07:03:23.430617 20917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91094b23ca054b2c8a2f14e1bb117621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } } }
I20260927 07:03:23.430780 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91094b23ca054b2c8a2f14e1bb117621. Latest consensus state: current_term: 1 leader_uuid: "91094b23ca054b2c8a2f14e1bb117621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91094b23ca054b2c8a2f14e1bb117621" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 37025 } } }
I20260927 07:03:23.431164 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:23.431303 20917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621 [sys.catalog]: This master's current role is: LEADER
W20260927 07:03:23.431906 20931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:23.431974 20931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:23.432060 20932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:23.432258 20932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:23.433881 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.427366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.3.57.62:37025)
0927 07:03:23.427627 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.427634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.427659 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:03:23.428242 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.428249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.428307 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.428427 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.428691 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.430367 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.431573 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.432155 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.432267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.433066 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.433069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.433069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:23.433246 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.433279 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:23.433289 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.433374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.433596 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.433680 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":110,"threads_started":1}
I20260927 07:03:23.434101 20932 catalog_manager.cc:1347] Generated new cluster ID: 192761877a8f43ed80fb2d7d1cccaa79
I20260927 07:03:23.434176 20932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:23.442790 20932 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:23.443460 20932 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:23.451367 20932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: Generated new TSK 0
I20260927 07:03:23.451525 20932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:23.454206 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--builtin_ntp_servers=127.3.57.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:23.532737 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:23.532920 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:23.532941 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:23.534060 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:23.534096 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:23.534121 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:23.535454 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:23.535703 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:23.536018 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:23.540792 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.540771 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.540778 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.541023 20935 server_base.cc:1034] running on GCE node
I20260927 07:03:23.541275 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:23.541507 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:23.542670 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790492603542656 us; error 28 us; skew 500 ppm
I20260927 07:03:23.544109 20935 webserver.cc:460] Webserver started at http://127.3.57.1:39835/ using document root <none> and password file <none>
I20260927 07:03:23.544432 20935 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:23.544507 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:23.544641 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:23.545971 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "21b890eb4ea54455a07156019822dbb9"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "3d03ee886c5da0180e8e98098a7e7b0e"
server_key_iv: "0f74b084e86b16bf119224b6ec86e6ed"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.546320 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "21b890eb4ea54455a07156019822dbb9"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "3d03ee886c5da0180e8e98098a7e7b0e"
server_key_iv: "0f74b084e86b16bf119224b6ec86e6ed"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.547674 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260927 07:03:23.548439 20949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.548594 20935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:23.548697 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "21b890eb4ea54455a07156019822dbb9"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "3d03ee886c5da0180e8e98098a7e7b0e"
server_key_iv: "0f74b084e86b16bf119224b6ec86e6ed"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.548766 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:23.572227 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:23.572566 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:23.572716 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:23.572942 20935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:23.573330 20935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:23.573413 20935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.573477 20935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:23.573503 20935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.579437 20935 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:43909
I20260927 07:03:23.579476 21062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:43909 every 8 connection(s)
I20260927 07:03:23.579797 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:23.584592 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.580489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:39401 (local address 127.3.57.62:37025)
0927 07:03:23.580558 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.580560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.580771 (+ 211us) server_negotiation.cc:407] Connection header received
0927 07:03:23.581224 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.581225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.581234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.581293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.581890 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.582378 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.583283 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.583460 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.583508 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.584315 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.584316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.584316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:23.584320 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.584349 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:23.584352 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.584396 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.584484 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.584532 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:03:23.584769 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.580487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37025 (local address 127.3.57.1:39401)
0927 07:03:23.580652 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.580691 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.581149 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.581342 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.581344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.581352 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:23.581826 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.581829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.582464 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.582465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.583205 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.583207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.583529 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.583529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.583686 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.584183 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:23.584183 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:23.584210 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:23.584231 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:23.584413 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:23.584418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:23.584420 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.584470 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.584571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260927 07:03:23.585340 21063 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37025
I20260927 07:03:23.585441 21063 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:23.585635 21063 heartbeater.cc:511] Master 127.3.57.62:37025 requested a full tablet report, sending...
I20260927 07:03:23.586117 20875 ts_manager.cc:194] Registered new tserver with Master: 21b890eb4ea54455a07156019822dbb9 (127.3.57.1:43909)
I20260927 07:03:23.586691 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.1:39401
I20260927 07:03:23.589062 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 20935
I20260927 07:03:23.589144 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:23.589358 3300 external_mini_cluster.cc:1428] Setting key 1729c4a246778a3224a4b223a0545124
I20260927 07:03:23.590334 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--builtin_ntp_servers=127.3.57.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:23.668830 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:23.669025 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:23.669047 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:23.670197 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:23.670236 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:23.670260 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:23.671600 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:23.671844 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:23.672127 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:23.676879 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.677043 21066 server_base.cc:1034] running on GCE node
W20260927 07:03:23.676872 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.676885 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.677361 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:23.677605 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:23.678750 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790492603678728 us; error 36 us; skew 500 ppm
I20260927 07:03:23.680016 21066 webserver.cc:460] Webserver started at http://127.3.57.2:36513/ using document root <none> and password file <none>
I20260927 07:03:23.680209 21066 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:23.680249 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:23.680337 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:23.681324 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "23a38b3506a3435cb44d7b4d2e31c5fc"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "0a414d9708478312616c420620241aac"
server_key_iv: "8cc2cb9d094e84d614adba2aaa7c370c"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.681625 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "23a38b3506a3435cb44d7b4d2e31c5fc"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "0a414d9708478312616c420620241aac"
server_key_iv: "8cc2cb9d094e84d614adba2aaa7c370c"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.682837 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:03:23.683501 21080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.683676 21066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.683772 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "23a38b3506a3435cb44d7b4d2e31c5fc"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "0a414d9708478312616c420620241aac"
server_key_iv: "8cc2cb9d094e84d614adba2aaa7c370c"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.683853 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:23.698465 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:23.698747 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:23.698897 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:23.699122 21066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:23.699438 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:23.699497 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.699537 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:23.699577 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.705401 21066 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46653
I20260927 07:03:23.705447 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46653 every 8 connection(s)
I20260927 07:03:23.705781 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:23.710692 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.706531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:37343 (local address 127.3.57.62:37025)
0927 07:03:23.706597 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.706599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.706834 (+ 235us) server_negotiation.cc:407] Connection header received
0927 07:03:23.707297 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.707298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.707307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.707364 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.708017 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.708509 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.709469 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.709648 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.709698 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.710466 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.710467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.710467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:23.710472 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.710487 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:23.710490 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.710531 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.710610 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.710631 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:03:23.710878 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.706501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37025 (local address 127.3.57.2:37343)
0927 07:03:23.706708 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.706748 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.707219 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.707424 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.707427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.707435 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:23.707944 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.707948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.708592 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.708593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.709375 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.709377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.709721 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.709722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.709826 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.710324 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:23.710325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:23.710351 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:23.710373 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:23.710543 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:23.710547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:23.710549 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.710600 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.710678 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260927 07:03:23.711359 21194 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37025
I20260927 07:03:23.711464 21194 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:23.711623 21194 heartbeater.cc:511] Master 127.3.57.62:37025 requested a full tablet report, sending...
I20260927 07:03:23.711992 20875 ts_manager.cc:194] Registered new tserver with Master: 23a38b3506a3435cb44d7b4d2e31c5fc (127.3.57.2:46653)
I20260927 07:03:23.712359 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.2:37343
I20260927 07:03:23.714950 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21066
I20260927 07:03:23.715040 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:23.715185 3300 external_mini_cluster.cc:1428] Setting key 206b67bd226da9384b46682c0a0e3086
I20260927 07:03:23.716080 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--builtin_ntp_servers=127.3.57.20:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:23.797262 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:23.797441 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:23.797461 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:23.798556 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:23.798595 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:23.798647 21197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:23.800028 21197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:23.800269 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:23.800563 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:23.805346 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.805439 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:23.805476 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:23.805439 21197 server_base.cc:1034] running on GCE node
I20260927 07:03:23.805776 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:23.806020 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:23.807178 21197 hybrid_clock.cc:648] HybridClock initialized: now 1790492603807174 us; error 36 us; skew 500 ppm
I20260927 07:03:23.808386 21197 webserver.cc:460] Webserver started at http://127.3.57.3:36215/ using document root <none> and password file <none>
I20260927 07:03:23.808624 21197 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:23.808691 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:23.808833 21197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:23.809821 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "d945a9469c2648db91726bf6900ee930"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "a7ba0cf519b3d3a6448f62ee5944baac"
server_key_iv: "85d2183755776def50ae6882e62870ad"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.810164 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "d945a9469c2648db91726bf6900ee930"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "a7ba0cf519b3d3a6448f62ee5944baac"
server_key_iv: "85d2183755776def50ae6882e62870ad"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.811347 21197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:23.812057 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.812219 21197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.812305 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "d945a9469c2648db91726bf6900ee930"
format_stamp: "Formatted at 2026-09-27 07:03:23 on dist-test-slave-mk3h"
server_key: "a7ba0cf519b3d3a6448f62ee5944baac"
server_key_iv: "85d2183755776def50ae6882e62870ad"
server_key_version: "encryptionkey@0"
I20260927 07:03:23.812382 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:23.827005 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:23.827301 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:23.827441 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:23.827723 21197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:23.828040 21197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:23.828094 21197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.828135 21197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:23.828161 21197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:23.834033 21197 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:36005
I20260927 07:03:23.834105 21324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:36005 every 8 connection(s)
I20260927 07:03:23.834429 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:23.839653 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.835179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:45361 (local address 127.3.57.62:37025)
0927 07:03:23.835285 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.835287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.835747 (+ 460us) server_negotiation.cc:407] Connection header received
0927 07:03:23.836236 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.836237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.836247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.836301 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.836919 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.837428 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.838372 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.838551 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.838602 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.839341 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.839342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.839342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:23.839346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.839359 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:23.839362 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.839404 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.839499 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.839519 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:03:23.839820 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.835266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:37025 (local address 127.3.57.3:45361)
0927 07:03:23.835479 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.835513 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.836141 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.836352 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.836355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.836363 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:23.836839 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.836843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.837513 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.837514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.838297 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.838299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.838623 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.838624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.838728 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.839206 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:23.839206 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:23.839233 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:23.839254 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:23.839419 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:23.839423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:23.839425 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.839475 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.839602 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260927 07:03:23.840395 21325 heartbeater.cc:347] Connected to a master server at 127.3.57.62:37025
I20260927 07:03:23.840523 21325 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:23.840626 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21197
I20260927 07:03:23.840711 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:23.840734 21325 heartbeater.cc:511] Master 127.3.57.62:37025 requested a full tablet report, sending...
I20260927 07:03:23.840941 3300 external_mini_cluster.cc:1428] Setting key 8d9026df3399f98c6ea548c4736e9086
I20260927 07:03:23.841135 20875 ts_manager.cc:194] Registered new tserver with Master: d945a9469c2648db91726bf6900ee930 (127.3.57.3:36005)
I20260927 07:03:23.841627 20875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.3:45361
I20260927 07:03:23.841857 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:23.846019 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.843201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.3.57.62:37025)
0927 07:03:23.843275 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.843277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.843352 (+ 75us) server_negotiation.cc:407] Connection header received
0927 07:03:23.843425 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.843426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.843435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.843490 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:23.843700 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.844157 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.844917 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.845121 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.845195 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.845776 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:23.845777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:23.845777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:23.845781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:23.845793 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:23.845797 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:23.845863 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.845916 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.845935 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:03:23.847295 20875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:23.848841 20875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:23.855141 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:43909 (local address 127.0.0.1:40402)
0927 07:03:23.851323 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.851366 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.851437 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.851742 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.851745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.851747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.851961 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.851963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.852869 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.852870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.854020 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.854023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.854852 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.854853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.854931 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.854944 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.855022 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.855070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260927 07:03:23.855170 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.3.57.1:43909)
0927 07:03:23.851329 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.851334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.851518 (+ 184us) server_negotiation.cc:407] Connection header received
0927 07:03:23.851584 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.851586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.851602 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.851669 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.852038 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.852788 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.854107 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.854759 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.854834 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.854920 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.855042 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.855084 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260927 07:03:23.855490 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36005 (local address 127.0.0.1:44550)
0927 07:03:23.851474 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.851482 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.851537 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.851773 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.851774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.851776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.851968 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.851969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.853247 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.853248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.854272 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.854273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.855243 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.855244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.855306 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.855316 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.855388 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.855424 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":47,"thread_start_us":71,"threads_started":1}
I20260927 07:03:23.855520 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.3.57.3:36005)
0927 07:03:23.851642 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.851646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.851655 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:23.851689 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.851690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.851704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.851772 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.852052 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.853157 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.854350 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.855063 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.855126 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.855207 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.855406 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.855445 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":74,"threads_started":1}
I20260927 07:03:23.856326 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46653 (local address 127.0.0.1:40980)
0927 07:03:23.851346 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.851378 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.851436 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.851775 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.851776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.851777 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.851963 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.851966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.853258 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.853259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.854848 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.854849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.856102 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.856103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.856166 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.856169 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.856227 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.856265 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":849,"thread_start_us":54,"threads_started":1}
I20260927 07:03:23.856199 20997 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfcd460e9da432dbc602d54f151df44 (DEFAULT_TABLE table=test-table [id=eaa82b98dfa74e45aeaf0361cb0453b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.856386 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.851140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.3.57.2:46653)
0927 07:03:23.851409 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.851413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.851423 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:23.851533 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.851536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.851573 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.851655 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.852053 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.853167 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.854930 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.855740 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.855820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.855904 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.856260 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.856301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":88,"threads_started":1}
I20260927 07:03:23.856382 21259 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfcd460e9da432dbc602d54f151df44 (DEFAULT_TABLE table=test-table [id=eaa82b98dfa74e45aeaf0361cb0453b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.856577 20997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfcd460e9da432dbc602d54f151df44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.856772 21259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfcd460e9da432dbc602d54f151df44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.857220 21128 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfcd460e9da432dbc602d54f151df44 (DEFAULT_TABLE table=test-table [id=eaa82b98dfa74e45aeaf0361cb0453b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:23.857524 21128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfcd460e9da432dbc602d54f151df44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:23.860013 21340 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Bootstrap starting.
I20260927 07:03:23.860160 21339 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: Bootstrap starting.
I20260927 07:03:23.860658 21340 tablet_bootstrap.cc:654] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.860750 21341 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Bootstrap starting.
I20260927 07:03:23.860805 21339 tablet_bootstrap.cc:654] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.861101 21339 log.cc:824] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.861102 21340 log.cc:824] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.861415 21341 tablet_bootstrap.cc:654] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:23.861701 21341 log.cc:824] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:23.861850 21340 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: No bootstrap required, opened a new log
I20260927 07:03:23.861948 21340 ts_tablet_manager.cc:1397] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:23.862347 21339 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: No bootstrap required, opened a new log
I20260927 07:03:23.862421 21339 ts_tablet_manager.cc:1397] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:23.862460 21341 tablet_bootstrap.cc:492] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: No bootstrap required, opened a new log
I20260927 07:03:23.862558 21341 ts_tablet_manager.cc:1397] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:23.864033 21340 raft_consensus.cc:348] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.864184 21340 raft_consensus.cc:374] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.864244 21340 raft_consensus.cc:729] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d945a9469c2648db91726bf6900ee930, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.864382 21340 consensus_queue.cc:260] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.864672 21325 heartbeater.cc:503] Master 127.3.57.62:37025 was elected leader, sending a full tablet report...
I20260927 07:03:23.864656 21340 ts_tablet_manager.cc:1428] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 07:03:23.865080 21341 raft_consensus.cc:348] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.865075 21339 raft_consensus.cc:348] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.865228 21339 raft_consensus.cc:374] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.865270 21341 raft_consensus.cc:374] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:23.865291 21339 raft_consensus.cc:729] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b890eb4ea54455a07156019822dbb9, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.865320 21341 raft_consensus.cc:729] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a38b3506a3435cb44d7b4d2e31c5fc, State: Initialized, Role: FOLLOWER
I20260927 07:03:23.865410 21339 consensus_queue.cc:260] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.865460 21341 consensus_queue.cc:260] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.865701 21339 ts_tablet_manager.cc:1428] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:23.865701 21341 ts_tablet_manager.cc:1428] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:23.865823 21063 heartbeater.cc:503] Master 127.3.57.62:37025 was elected leader, sending a full tablet report...
I20260927 07:03:23.865844 21194 heartbeater.cc:503] Master 127.3.57.62:37025 was elected leader, sending a full tablet report...
I20260927 07:03:23.878695 21347 raft_consensus.cc:482] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:23.878774 21347 raft_consensus.cc:504] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.879140 21347 leader_election.cc:288] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a38b3506a3435cb44d7b4d2e31c5fc (127.3.57.2:46653), d945a9469c2648db91726bf6900ee930 (127.3.57.3:36005)
I20260927 07:03:23.882988 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.879370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:35181 (local address 127.3.57.2:46653)
0927 07:03:23.879413 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.879415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.879425 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:23.879477 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.879478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.879486 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.879580 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.879734 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.880495 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882051 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.882598 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882673 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.882682 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.882891 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.882927 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:03:23.882988 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.879366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:39901 (local address 127.3.57.3:36005)
0927 07:03:23.879418 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.879420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.879606 (+ 186us) server_negotiation.cc:407] Connection header received
0927 07:03:23.879669 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.879670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.879677 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.879742 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:23.879837 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.880510 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882051 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.882599 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.882682 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.882885 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.882921 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:03:23.883015 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.879311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46653 (local address 127.3.57.1:35181)
0927 07:03:23.879366 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.879378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.879420 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.879592 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.879593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.879595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.879666 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.879668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.880605 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.880607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.881965 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.881967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.882701 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.882802 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.882804 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.882867 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.882921 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":88}
I20260927 07:03:23.883023 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.879326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:36005 (local address 127.3.57.1:39901)
0927 07:03:23.879486 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.879495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:23.879606 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:23.879742 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:23.879743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:23.879744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:23.879797 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.879798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.880605 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.880607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.881962 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:23.881965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.882701 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.882702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:23.882799 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.882803 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:23.882867 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 07:03:23.882920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":43,"thread_start_us":94,"threads_started":1}
I20260927 07:03:23.883330 21148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfcd460e9da432dbc602d54f151df44" candidate_uuid: "21b890eb4ea54455a07156019822dbb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" is_pre_election: true
I20260927 07:03:23.883386 21279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfcd460e9da432dbc602d54f151df44" candidate_uuid: "21b890eb4ea54455a07156019822dbb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d945a9469c2648db91726bf6900ee930" is_pre_election: true
I20260927 07:03:23.883472 21148 raft_consensus.cc:2454] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b890eb4ea54455a07156019822dbb9 in term 0.
I20260927 07:03:23.883490 21279 raft_consensus.cc:2454] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b890eb4ea54455a07156019822dbb9 in term 0.
I20260927 07:03:23.883730 20952 leader_election.cc:302] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b890eb4ea54455a07156019822dbb9, 23a38b3506a3435cb44d7b4d2e31c5fc; no voters:
I20260927 07:03:23.883893 21347 raft_consensus.cc:2781] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:23.883962 21347 raft_consensus.cc:482] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:23.883981 21347 raft_consensus.cc:3037] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.884636 21347 raft_consensus.cc:504] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.884821 21347 leader_election.cc:288] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [CANDIDATE]: Term 1 election: Requested vote from peers 23a38b3506a3435cb44d7b4d2e31c5fc (127.3.57.2:46653), d945a9469c2648db91726bf6900ee930 (127.3.57.3:36005)
I20260927 07:03:23.884897 21279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfcd460e9da432dbc602d54f151df44" candidate_uuid: "21b890eb4ea54455a07156019822dbb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d945a9469c2648db91726bf6900ee930"
I20260927 07:03:23.884970 21279 raft_consensus.cc:3037] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.885659 21279 raft_consensus.cc:2454] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b890eb4ea54455a07156019822dbb9 in term 1.
I20260927 07:03:23.885771 21148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfcd460e9da432dbc602d54f151df44" candidate_uuid: "21b890eb4ea54455a07156019822dbb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc"
I20260927 07:03:23.885819 20951 leader_election.cc:302] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b890eb4ea54455a07156019822dbb9, d945a9469c2648db91726bf6900ee930; no voters:
I20260927 07:03:23.885866 21148 raft_consensus.cc:3037] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:23.885915 21347 raft_consensus.cc:2781] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:23.886068 21347 raft_consensus.cc:686] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [term 1 LEADER]: Becoming Leader. State: Replica: 21b890eb4ea54455a07156019822dbb9, State: Running, Role: LEADER
I20260927 07:03:23.886162 21347 consensus_queue.cc:237] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } }
I20260927 07:03:23.886711 21148 raft_consensus.cc:2454] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b890eb4ea54455a07156019822dbb9 in term 1.
I20260927 07:03:23.886890 20874 catalog_manager.cc:5495] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b890eb4ea54455a07156019822dbb9 (127.3.57.1). New cstate: current_term: 1 leader_uuid: "21b890eb4ea54455a07156019822dbb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b890eb4ea54455a07156019822dbb9" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 } health_report { overall_health: UNKNOWN } } }
I20260927 07:03:23.899467 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.897328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.3.57.1:43909)
0927 07:03:23.897366 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:23.897368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:23.897376 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:23.897457 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:23.897457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:23.897471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:23.897528 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:23.897619 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.898042 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.898817 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:23.898991 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:23.899038 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:23.899189 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:23.899273 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:23.899329 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:23.899388 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 07:03:23.899404 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:23.901031 21279 raft_consensus.cc:1261] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 1 FOLLOWER]: Refusing update from remote peer 21b890eb4ea54455a07156019822dbb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:23.901054 21148 raft_consensus.cc:1261] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 1 FOLLOWER]: Refusing update from remote peer 21b890eb4ea54455a07156019822dbb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:23.901343 21347 consensus_queue.cc:1035] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d945a9469c2648db91726bf6900ee930" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:23.901505 21349 consensus_queue.cc:1035] T 8bfcd460e9da432dbc602d54f151df44 P 21b890eb4ea54455a07156019822dbb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a38b3506a3435cb44d7b4d2e31c5fc" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:23.904165 21353 mvcc.cc:205] Tried to move back new op lower bound from 7333857705576394752 to 7333857705518313472. Current Snapshot: MvccSnapshot[applied={T|T < 7333857705576394752}]
I20260927 07:03:23.904290 21354 mvcc.cc:205] Tried to move back new op lower bound from 7333857705576394752 to 7333857705518313472. Current Snapshot: MvccSnapshot[applied={T|T < 7333857705576394752}]
I20260927 07:03:23.905782 20875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:23.905865 20875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:23.906612 20875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91094b23ca054b2c8a2f14e1bb117621: Sending DeleteTablet for 3 replicas of tablet 8bfcd460e9da432dbc602d54f151df44
I20260927 07:03:23.906953 21128 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfcd460e9da432dbc602d54f151df44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='slave'} at 127.0.0.1:40980
I20260927 07:03:23.907011 20997 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfcd460e9da432dbc602d54f151df44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='slave'} at 127.0.0.1:40402
I20260927 07:03:23.907138 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20935
I20260927 07:03:23.907160 21259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfcd460e9da432dbc602d54f151df44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:23 UTC) from {username='slave'} at 127.0.0.1:44550
I20260927 07:03:23.908102 21363 tablet_replica.cc:330] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: stopping tablet replica
I20260927 07:03:23.908203 21363 raft_consensus.cc:2229] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:23.908305 21363 raft_consensus.cc:2258] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:23.908654 21363 ts_tablet_manager.cc:1905] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:23.910152 21361 tablet_replica.cc:330] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: stopping tablet replica
I20260927 07:03:23.910288 21361 raft_consensus.cc:2229] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:23.910327 21363 ts_tablet_manager.cc:1918] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:23.910403 21363 log.cc:1196] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/8bfcd460e9da432dbc602d54f151df44
I20260927 07:03:23.910405 21361 raft_consensus.cc:2258] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:23.910764 21363 ts_tablet_manager.cc:1939] T 8bfcd460e9da432dbc602d54f151df44 P d945a9469c2648db91726bf6900ee930: Deleting consensus metadata
I20260927 07:03:23.910763 21361 ts_tablet_manager.cc:1905] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:23.911175 20860 catalog_manager.cc:4841] TS d945a9469c2648db91726bf6900ee930 (127.3.57.3:36005): tablet 8bfcd460e9da432dbc602d54f151df44 (table test-table [id=eaa82b98dfa74e45aeaf0361cb0453b0]) successfully deleted
I20260927 07:03:23.912633 21361 ts_tablet_manager.cc:1918] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:23.912707 21361 log.cc:1196] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/8bfcd460e9da432dbc602d54f151df44
I20260927 07:03:23.912948 21361 ts_tablet_manager.cc:1939] T 8bfcd460e9da432dbc602d54f151df44 P 23a38b3506a3435cb44d7b4d2e31c5fc: Deleting consensus metadata
I20260927 07:03:23.913434 20861 catalog_manager.cc:4841] TS 23a38b3506a3435cb44d7b4d2e31c5fc (127.3.57.2:46653): tablet 8bfcd460e9da432dbc602d54f151df44 (table test-table [id=eaa82b98dfa74e45aeaf0361cb0453b0]) successfully deleted
W20260927 07:03:23.914554 20860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:43909 (error 108)
I20260927 07:03:23.914772 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21066
I20260927 07:03:23.915356 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:23.914895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:43909 (local address 127.0.0.1:40424)
0927 07:03:23.915278 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:23.915317 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
W20260927 07:03:23.915613 20860 catalog_manager.cc:4584] TS 21b890eb4ea54455a07156019822dbb9 (127.3.57.1:43909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bfcd460e9da432dbc602d54f151df44: Network error: Client connection negotiation failed: client connection to 127.3.57.1:43909: connect: Connection refused (error 111)
I20260927 07:03:23.919394 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21197
I20260927 07:03:23.923766 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 20844
2026-09-27T07:03:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (648 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:23 dist-test-slave-mk3h krb5kdc[21367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:23 dist-test-slave-mk3h krb5kdc[21367](info): set up 2 sockets
Sep 27 07:03:23 dist-test-slave-mk3h krb5kdc[21367](info): commencing operation
krb5kdc: starting...
W20260927 07:03:24.963301 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:25.073745 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41083
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:25.152465 21380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:25.152688 21380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:25.152722 21380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:25.153844 21380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:25.153915 21380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:25.153944 21380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:25.153988 21380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:25.154016 21380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:25.155438 21380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:25.155706 21380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:25.155927 21380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:25.160450 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.160434 21385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.160506 21386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.160555 21380 server_base.cc:1034] running on GCE node
I20260927 07:03:25.160987 21380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:25.161265 21380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:25.162436 21380 hybrid_clock.cc:648] HybridClock initialized: now 1790492605162437 us; error 42 us; skew 500 ppm
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:25.165017 21380 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:25.165450 21380 webserver.cc:460] Webserver started at http://127.3.57.62:43979/ using document root <none> and password file <none>
I20260927 07:03:25.165620 21380 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:25.165658 21380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:25.165741 21380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:25.166666 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "f378d6c3244b4530a4e5aa2b642e61a9"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "aa1f09acb98c10620ad179dc4de14f8a"
server_key_iv: "bbb8150e7585940fafcf5f2041d72756"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.166955 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "f378d6c3244b4530a4e5aa2b642e61a9"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "aa1f09acb98c10620ad179dc4de14f8a"
server_key_iv: "bbb8150e7585940fafcf5f2041d72756"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.168237 21380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:25.168885 21395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.169075 21380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:25.169142 21380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "f378d6c3244b4530a4e5aa2b642e61a9"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "aa1f09acb98c10620ad179dc4de14f8a"
server_key_iv: "bbb8150e7585940fafcf5f2041d72756"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.169193 21380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:25.182431 21380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:25.184508 21380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:25.184609 21380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:25.188556 21380 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41083
I20260927 07:03:25.188627 21447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41083 every 8 connection(s)
I20260927 07:03:25.188915 21380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:25.189700 21448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:25.191802 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: Bootstrap starting.
I20260927 07:03:25.192303 21448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:25.192534 21448 log.cc:824] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:25.193159 21448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: No bootstrap required, opened a new log
I20260927 07:03:25.194656 21448 raft_consensus.cc:348] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } }
I20260927 07:03:25.194782 21448 raft_consensus.cc:374] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:25.194803 21448 raft_consensus.cc:729] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f378d6c3244b4530a4e5aa2b642e61a9, State: Initialized, Role: FOLLOWER
I20260927 07:03:25.194900 21448 consensus_queue.cc:260] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } }
I20260927 07:03:25.194978 21448 raft_consensus.cc:388] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:25.195024 21448 raft_consensus.cc:482] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:25.195096 21448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:25.195760 21448 raft_consensus.cc:504] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } }
I20260927 07:03:25.195868 21448 leader_election.cc:302] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f378d6c3244b4530a4e5aa2b642e61a9; no voters:
I20260927 07:03:25.196062 21448 leader_election.cc:288] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:25.196143 21451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:25.196297 21451 raft_consensus.cc:686] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [term 1 LEADER]: Becoming Leader. State: Replica: f378d6c3244b4530a4e5aa2b642e61a9, State: Running, Role: LEADER
I20260927 07:03:25.196317 21448 sys_catalog.cc:564] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:25.196380 21451 consensus_queue.cc:237] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } }
I20260927 07:03:25.196954 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } } }
I20260927 07:03:25.197052 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:25.197161 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f378d6c3244b4530a4e5aa2b642e61a9. Latest consensus state: current_term: 1 leader_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f378d6c3244b4530a4e5aa2b642e61a9" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41083 } } }
I20260927 07:03:25.197338 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:25.197567 21461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:25.198043 21461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:25.198108 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21380
I20260927 07:03:25.198194 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:25.198333 3300 external_mini_cluster.cc:1428] Setting key 8035238693a63a4820fb53f667cb65a0
I20260927 07:03:25.199776 21461 catalog_manager.cc:1347] Generated new cluster ID: 2403f0bbc79f4097af8b93b2274674f0
I20260927 07:03:25.199831 21461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:25.209318 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.198711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.3.57.62:41083)
0927 07:03:25.198892 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.198901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.198936 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:03:25.199075 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.199081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.199153 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.199289 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.200061 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.201377 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.202407 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.202994 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.203073 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.205725 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.205728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.205728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:25.205740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.207602 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.207951 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.207954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.207954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.208045 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.208215 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.208215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.208216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.208471 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.208580 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.208957 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.209081 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260927 07:03:25.211869 21461 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:25.212360 21461 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:25.216696 21461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: Generated new TSK 0
I20260927 07:03:25.216867 21461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:03:25.244390 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:25.334570 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:25.334750 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:25.334771 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:25.335922 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:25.335961 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:25.335986 21474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:25.337347 21474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:25.337577 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:25.337791 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:25.343468 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.343575 21474 server_base.cc:1034] running on GCE node
W20260927 07:03:25.343606 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.343652 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.343847 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:25.344085 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:25.345232 21474 hybrid_clock.cc:648] HybridClock initialized: now 1790492605345213 us; error 39 us; skew 500 ppm
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:25.347688 21474 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:03:25.348135 21474 webserver.cc:460] Webserver started at http://127.3.57.1:43635/ using document root <none> and password file <none>
I20260927 07:03:25.348364 21474 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:25.348433 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:25.348562 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:25.349643 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "b6a3af223df84bfda29789844c01b1bc"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "f53efcf720e3ed8e538d4e49ff0a5737"
server_key_iv: "d2f936ae5be5f88b8931ff1d91e6fae2"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.350003 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "b6a3af223df84bfda29789844c01b1bc"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "f53efcf720e3ed8e538d4e49ff0a5737"
server_key_iv: "d2f936ae5be5f88b8931ff1d91e6fae2"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.351298 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:03:25.352154 21489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.352368 21474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:25.352463 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "b6a3af223df84bfda29789844c01b1bc"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "f53efcf720e3ed8e538d4e49ff0a5737"
server_key_iv: "d2f936ae5be5f88b8931ff1d91e6fae2"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.352547 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:25.366240 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:25.368202 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:25.368362 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:25.368587 21474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:25.368878 21474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:25.368930 21474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.368965 21474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:25.368996 21474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.374985 21474 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44083
I20260927 07:03:25.375036 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44083 every 8 connection(s)
I20260927 07:03:25.375339 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:25.379263 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21474
I20260927 07:03:25.379407 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:25.379642 3300 external_mini_cluster.cc:1428] Setting key df14d6dd0ac9c7a479a76463d5207d1d
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:25.385142 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.376032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41083 (local address 127.3.57.1:38483)
0927 07:03:25.376239 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.376277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.376776 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.377256 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.377259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.377821 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:25.378014 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.378018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.378724 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.378724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.379655 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.379659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.380104 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.380105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.380268 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.380830 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:25.380843 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:25.382135 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:25.383931 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.383936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.383938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.384141 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.384345 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.384346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.384346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.384435 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.384707 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:25.384711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:25.384810 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.384885 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.384993 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260927 07:03:25.385183 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.376049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:38483 (local address 127.3.57.62:41083)
0927 07:03:25.376118 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.376121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.376392 (+ 271us) server_negotiation.cc:407] Connection header received
0927 07:03:25.376992 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.376993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.377004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.377052 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.378118 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.378644 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.379739 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.380011 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.380094 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.382229 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.382231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.382231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:25.382238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.383848 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.384215 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.384216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.384217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.384277 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.384498 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.384499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.384500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.384641 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.384715 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.384941 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.385036 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:03:25.385771 21603 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41083
I20260927 07:03:25.385865 21603 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:25.386045 21603 heartbeater.cc:511] Master 127.3.57.62:41083 requested a full tablet report, sending...
I20260927 07:03:25.386523 21412 ts_manager.cc:194] Registered new tserver with Master: b6a3af223df84bfda29789844c01b1bc (127.3.57.1:44083)
I20260927 07:03:25.387127 21412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:38483
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:03:25.406059 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:25.488085 21608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:25.488329 21608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:25.488368 21608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:25.489538 21608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:25.489606 21608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:25.489651 21608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:25.491097 21608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:25.491365 21608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:25.491600 21608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:25.496258 21616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.496424 21613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.496449 21614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.496687 21608 server_base.cc:1034] running on GCE node
I20260927 07:03:25.496906 21608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:25.497166 21608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:25.498312 21608 hybrid_clock.cc:648] HybridClock initialized: now 1790492605498297 us; error 28 us; skew 500 ppm
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:25.500723 21608 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:03:25.501170 21608 webserver.cc:460] Webserver started at http://127.3.57.2:43017/ using document root <none> and password file <none>
I20260927 07:03:25.501401 21608 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:25.501466 21608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:25.501613 21608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:25.502575 21608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "2f4df10e30ed430e910f9cf855fcb4ad"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "b93c6d4373dade5da5d8d4a9651646de"
server_key_iv: "98150f78dc2871bcad2a57712350b52f"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.502915 21608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "2f4df10e30ed430e910f9cf855fcb4ad"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "b93c6d4373dade5da5d8d4a9651646de"
server_key_iv: "98150f78dc2871bcad2a57712350b52f"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.504236 21608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:25.504895 21623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.505081 21608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:25.505179 21608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "2f4df10e30ed430e910f9cf855fcb4ad"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "b93c6d4373dade5da5d8d4a9651646de"
server_key_iv: "98150f78dc2871bcad2a57712350b52f"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.505254 21608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:25.543074 21608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:25.544965 21608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:25.545118 21608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:25.545341 21608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:25.545634 21608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:25.545686 21608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.545727 21608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:25.545753 21608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.552207 21608 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:46439
I20260927 07:03:25.552294 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:46439 every 8 connection(s)
I20260927 07:03:25.552599 21608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:25.561005 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21608
I20260927 07:03:25.561117 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:25.561307 3300 external_mini_cluster.cc:1428] Setting key 9316476959f0f4778ff2fe834f3c6cf4
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:25.569583 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.559450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:33823 (local address 127.3.57.62:41083)
0927 07:03:25.559505 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.559507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.559745 (+ 238us) server_negotiation.cc:407] Connection header received
0927 07:03:25.560361 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.560362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.560371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.560426 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.561601 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.562200 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.563734 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.564046 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.564123 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.566415 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.566416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.566417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:25.566424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.568097 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.568615 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.568616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.568617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.568681 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.568934 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.568935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.568935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.569047 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.569123 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.569410 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.569494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:03:25.569768 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.559404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41083 (local address 127.3.57.2:33823)
0927 07:03:25.559608 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.559653 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.560259 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.560485 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.560488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.561210 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:25.561507 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.561512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.562289 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.562290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.563659 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.563661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.564130 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.564131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.564230 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.564742 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:25.564752 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:25.566318 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:25.568198 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.568205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.568208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.568530 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.568746 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.568747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.568748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.568859 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.569113 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:25.569121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:25.569267 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.569370 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.569522 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
I20260927 07:03:25.570317 21737 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41083
I20260927 07:03:25.570421 21737 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:25.570590 21737 heartbeater.cc:511] Master 127.3.57.62:41083 requested a full tablet report, sending...
I20260927 07:03:25.571007 21412 ts_manager.cc:194] Registered new tserver with Master: 2f4df10e30ed430e910f9cf855fcb4ad (127.3.57.2:46439)
I20260927 07:03:25.571384 21412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:33823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:03:25.585927 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:25.663162 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:25.663344 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:25.663365 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:25.664523 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:25.664564 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:25.664590 21742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:25.665921 21742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:25.666101 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:25.666257 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:25.671518 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.671581 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.671793 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.671802 21742 server_base.cc:1034] running on GCE node
I20260927 07:03:25.672055 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:25.672317 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:25.673484 21742 hybrid_clock.cc:648] HybridClock initialized: now 1790492605673465 us; error 33 us; skew 500 ppm
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:25.676074 21742 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:03:25.676543 21742 webserver.cc:460] Webserver started at http://127.3.57.3:37809/ using document root <none> and password file <none>
I20260927 07:03:25.676767 21742 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:25.676831 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:25.676962 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:25.677978 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "dfc7e9f260024bf2bfcd0e50a1639a87"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "7e0ab9d4be668b23219ba50b3d5e99d5"
server_key_iv: "e6d39b1d084ebbe23da6b519b94d5b08"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.678333 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "dfc7e9f260024bf2bfcd0e50a1639a87"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "7e0ab9d4be668b23219ba50b3d5e99d5"
server_key_iv: "e6d39b1d084ebbe23da6b519b94d5b08"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.679680 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:25.680369 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.680537 21742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:25.680631 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "dfc7e9f260024bf2bfcd0e50a1639a87"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "7e0ab9d4be668b23219ba50b3d5e99d5"
server_key_iv: "e6d39b1d084ebbe23da6b519b94d5b08"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.680716 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:25.726679 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:25.728539 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:25.728703 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:25.728921 21742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:25.729209 21742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:25.729295 21742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.729336 21742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:25.729363 21742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.735383 21742 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33109
I20260927 07:03:25.735481 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33109 every 8 connection(s)
I20260927 07:03:25.735754 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:25.740389 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21742
I20260927 07:03:25.740499 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:25.740645 3300 external_mini_cluster.cc:1428] Setting key 542093fe944ca1090bb18f211774b3ff
Sep 27 07:03:25 dist-test-slave-mk3h krb5kdc[21367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492605, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:25.746973 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.736796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:60999 (local address 127.3.57.62:41083)
0927 07:03:25.736839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.736841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.736853 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:25.737627 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.737628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.737639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.737726 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.738991 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.740046 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.741575 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.741907 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.741985 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.744556 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.744558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.744558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:25.744565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.745974 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.746291 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.746292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.746292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.746337 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.746531 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.746531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.746531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.746630 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.746674 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.746858 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.746907 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:03:25.747251 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.736534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41083 (local address 127.3.57.3:60999)
0927 07:03:25.736704 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.736753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.737490 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.737799 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.737803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.738615 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:25.738876 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.738882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.740206 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.740207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.741407 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.741410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.742056 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.742057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.742157 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.742645 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:25.742668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:25.744478 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:25.746041 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.746045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.746046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.746233 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.746391 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:25.746391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:25.746392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:25.746479 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:25.746679 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:25.746682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:25.746777 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.746848 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.746967 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260927 07:03:25.747706 21871 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41083
I20260927 07:03:25.747781 21871 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:25.747959 21871 heartbeater.cc:511] Master 127.3.57.62:41083 requested a full tablet report, sending...
I20260927 07:03:25.748329 21412 ts_manager.cc:194] Registered new tserver with Master: dfc7e9f260024bf2bfcd0e50a1639a87 (127.3.57.3:33109)
I20260927 07:03:25.748678 21412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:60999
I20260927 07:03:25.752318 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:25.759117 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.753609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.3.57.62:41083)
0927 07:03:25.753666 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.753667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.753750 (+ 83us) server_negotiation.cc:407] Connection header received
0927 07:03:25.753803 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.753803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.753811 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.753861 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.754347 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.754809 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.755532 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.755737 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.755785 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.756913 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.756914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.756915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:25.756920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.758225 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.758564 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.758565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.758565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.758608 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:25.758743 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:25.758743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:25.758743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:25.758843 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.758913 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.759023 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.759066 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 07:03:25.760816 21412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:25.762257 21412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:25.768576 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44083 (local address 127.0.0.1:60324)
0927 07:03:25.764837 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.764846 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.764906 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.765115 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.765118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.765120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:25.765212 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.765214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766457 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.766458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.767647 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.767649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768293 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.768369 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768382 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.768452 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.768503 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":52,"thread_start_us":73,"threads_started":1}
I20260927 07:03:25.768589 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.3.57.1:44083)
0927 07:03:25.764805 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.764810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.764920 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:03:25.764969 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.764971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.764988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.765188 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:25.765266 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766357 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.767727 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768201 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768293 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768353 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.768467 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.768507 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260927 07:03:25.768702 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46439 (local address 127.0.0.1:57316)
0927 07:03:25.764989 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.765000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.765037 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.765266 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.765267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.765268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:25.765306 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.765308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766516 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.766517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.767458 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.767460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768439 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.768504 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768517 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.768566 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.768611 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":8,"thread_start_us":106,"threads_started":1}
I20260927 07:03:25.769003 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.3.57.2:46439)
0927 07:03:25.765044 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.765049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.765058 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:25.765113 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.765115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.765127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.765194 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:25.765380 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766432 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.767570 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768270 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768400 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.768857 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.768904 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":75,"threads_started":1}
I20260927 07:03:25.769088 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.3.57.3:33109)
0927 07:03:25.764715 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.764721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.764812 (+ 91us) server_negotiation.cc:407] Connection header received
0927 07:03:25.764955 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.764958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.764975 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.765051 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:25.765371 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766052 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.767608 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768292 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768349 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768399 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.768605 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.769002 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
I20260927 07:03:25.769335 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.764372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33109 (local address 127.0.0.1:43682)
0927 07:03:25.764712 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.764750 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.764848 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.765135 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.765136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.765138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:25.765295 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.765296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.766152 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.766152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.767501 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.767504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.768438 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.768439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.768504 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.768514 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.769207 (+ 693us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.769274 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":74,"thread_start_us":53,"threads_started":1}
I20260927 07:03:25.769382 21537 tablet_service.cc:1467] Processing CreateTablet for tablet 30254dd424394dd8b7abc13b6d0262aa (DEFAULT_TABLE table=test-table [id=750be1f8d1c34dd18b8546b5e3e83301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:25.769743 21537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30254dd424394dd8b7abc13b6d0262aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:25.769644 21671 tablet_service.cc:1467] Processing CreateTablet for tablet 30254dd424394dd8b7abc13b6d0262aa (DEFAULT_TABLE table=test-table [id=750be1f8d1c34dd18b8546b5e3e83301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:25.769966 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30254dd424394dd8b7abc13b6d0262aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:25.769927 21805 tablet_service.cc:1467] Processing CreateTablet for tablet 30254dd424394dd8b7abc13b6d0262aa (DEFAULT_TABLE table=test-table [id=750be1f8d1c34dd18b8546b5e3e83301]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:25.770210 21805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30254dd424394dd8b7abc13b6d0262aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:25.772603 21888 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: Bootstrap starting.
I20260927 07:03:25.773236 21889 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Bootstrap starting.
I20260927 07:03:25.773384 21888 tablet_bootstrap.cc:654] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:25.773548 21890 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Bootstrap starting.
I20260927 07:03:25.773931 21889 tablet_bootstrap.cc:654] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:25.774085 21888 log.cc:824] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:25.774165 21890 tablet_bootstrap.cc:654] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:25.774497 21889 log.cc:824] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:25.774863 21890 log.cc:824] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:25.774977 21888 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: No bootstrap required, opened a new log
I20260927 07:03:25.775048 21888 ts_tablet_manager.cc:1397] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:03:25.775421 21889 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: No bootstrap required, opened a new log
I20260927 07:03:25.775532 21889 ts_tablet_manager.cc:1397] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:25.775704 21890 tablet_bootstrap.cc:492] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: No bootstrap required, opened a new log
I20260927 07:03:25.775795 21890 ts_tablet_manager.cc:1397] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:25.777014 21888 raft_consensus.cc:348] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.777197 21888 raft_consensus.cc:374] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:25.777237 21888 raft_consensus.cc:729] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a3af223df84bfda29789844c01b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:03:25.777371 21888 consensus_queue.cc:260] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.777520 21889 raft_consensus.cc:348] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.777644 21888 ts_tablet_manager.cc:1428] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:25.777678 21889 raft_consensus.cc:374] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:25.777714 21889 raft_consensus.cc:729] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4df10e30ed430e910f9cf855fcb4ad, State: Initialized, Role: FOLLOWER
I20260927 07:03:25.777753 21603 heartbeater.cc:503] Master 127.3.57.62:41083 was elected leader, sending a full tablet report...
I20260927 07:03:25.777652 21890 raft_consensus.cc:348] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.777866 21890 raft_consensus.cc:374] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:25.777841 21889 consensus_queue.cc:260] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.777916 21890 raft_consensus.cc:729] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc7e9f260024bf2bfcd0e50a1639a87, State: Initialized, Role: FOLLOWER
I20260927 07:03:25.778093 21889 ts_tablet_manager.cc:1428] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:25.778069 21890 consensus_queue.cc:260] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.778204 21737 heartbeater.cc:503] Master 127.3.57.62:41083 was elected leader, sending a full tablet report...
I20260927 07:03:25.778343 21890 ts_tablet_manager.cc:1428] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:03:25.778460 21871 heartbeater.cc:503] Master 127.3.57.62:41083 was elected leader, sending a full tablet report...
I20260927 07:03:25.779253 21896 raft_consensus.cc:482] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:25.779363 21896 raft_consensus.cc:504] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.779732 21896 leader_election.cc:288] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a3af223df84bfda29789844c01b1bc (127.3.57.1:44083), 2f4df10e30ed430e910f9cf855fcb4ad (127.3.57.2:46439)
I20260927 07:03:25.783442 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.779848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44083 (local address 127.3.57.3:56045)
0927 07:03:25.779895 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.779906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.779945 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.780161 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.780162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.780162 (+ 0us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:25.780221 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.780223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.781153 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.781153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.782410 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.782413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.783207 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.783208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.783296 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.783299 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.783347 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.783376 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":45}
I20260927 07:03:25.783453 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.779995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:51319 (local address 127.3.57.2:46439)
0927 07:03:25.780045 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.780047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.780061 (+ 14us) server_negotiation.cc:407] Connection header received
0927 07:03:25.780082 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.780083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.780094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.780150 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:25.780283 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.781065 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.782540 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.783096 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.783168 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.783180 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.783353 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.783391 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:03:25.783461 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.779948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:56045 (local address 127.3.57.1:44083)
0927 07:03:25.780045 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.780047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.780055 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:25.780080 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.780081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.780091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.780150 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:25.780295 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.781050 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.782499 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.783135 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.783195 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.783203 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.783370 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.783407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 07:03:25.783456 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.779865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:46439 (local address 127.3.57.3:51319)
0927 07:03:25.779979 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.779989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:25.780034 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:25.780160 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:25.780161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:25.780162 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:25.780222 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.780223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.781202 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.781203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.782432 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:25.782434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.783191 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.783192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:25.783263 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.783267 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:25.783320 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 07:03:25.783360 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260927 07:03:25.783805 21557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30254dd424394dd8b7abc13b6d0262aa" candidate_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a3af223df84bfda29789844c01b1bc" is_pre_election: true
I20260927 07:03:25.783926 21557 raft_consensus.cc:2454] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc7e9f260024bf2bfcd0e50a1639a87 in term 0.
I20260927 07:03:25.783907 21691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30254dd424394dd8b7abc13b6d0262aa" candidate_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" is_pre_election: true
I20260927 07:03:25.784016 21691 raft_consensus.cc:2454] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc7e9f260024bf2bfcd0e50a1639a87 in term 0.
I20260927 07:03:25.784173 21761 leader_election.cc:302] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6a3af223df84bfda29789844c01b1bc, dfc7e9f260024bf2bfcd0e50a1639a87; no voters:
I20260927 07:03:25.784298 21896 raft_consensus.cc:2781] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:25.784353 21896 raft_consensus.cc:482] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:25.784394 21896 raft_consensus.cc:3037] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:25.784976 21896 raft_consensus.cc:504] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.785264 21896 leader_election.cc:288] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [CANDIDATE]: Term 1 election: Requested vote from peers b6a3af223df84bfda29789844c01b1bc (127.3.57.1:44083), 2f4df10e30ed430e910f9cf855fcb4ad (127.3.57.2:46439)
I20260927 07:03:25.785315 21691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30254dd424394dd8b7abc13b6d0262aa" candidate_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4df10e30ed430e910f9cf855fcb4ad"
I20260927 07:03:25.785392 21691 raft_consensus.cc:3037] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:25.786046 21691 raft_consensus.cc:2454] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc7e9f260024bf2bfcd0e50a1639a87 in term 1.
I20260927 07:03:25.786167 21557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30254dd424394dd8b7abc13b6d0262aa" candidate_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a3af223df84bfda29789844c01b1bc"
I20260927 07:03:25.786255 21557 raft_consensus.cc:3037] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:25.786263 21760 leader_election.cc:302] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4df10e30ed430e910f9cf855fcb4ad, dfc7e9f260024bf2bfcd0e50a1639a87; no voters:
I20260927 07:03:25.786362 21896 raft_consensus.cc:2781] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:25.786516 21896 raft_consensus.cc:686] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 1 LEADER]: Becoming Leader. State: Replica: dfc7e9f260024bf2bfcd0e50a1639a87, State: Running, Role: LEADER
I20260927 07:03:25.786616 21896 consensus_queue.cc:237] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } }
I20260927 07:03:25.787058 21557 raft_consensus.cc:2454] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc7e9f260024bf2bfcd0e50a1639a87 in term 1.
I20260927 07:03:25.787358 21411 catalog_manager.cc:5495] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfc7e9f260024bf2bfcd0e50a1639a87 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc7e9f260024bf2bfcd0e50a1639a87" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 } health_report { overall_health: UNKNOWN } } }
I20260927 07:03:25.793743 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.791131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.3.57.3:33109)
0927 07:03:25.791165 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.791167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.791176 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:25.791254 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.791255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.791274 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.791368 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:25.791506 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.792028 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.792915 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.793118 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.793206 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.793394 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:25.793485 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:25.793594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.793634 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.793642 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:03:25.795384 21691 raft_consensus.cc:1261] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 1 FOLLOWER]: Refusing update from remote peer dfc7e9f260024bf2bfcd0e50a1639a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:25.795398 21557 raft_consensus.cc:1261] T 30254dd424394dd8b7abc13b6d0262aa P b6a3af223df84bfda29789844c01b1bc [term 1 FOLLOWER]: Refusing update from remote peer dfc7e9f260024bf2bfcd0e50a1639a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:25.795743 21898 consensus_queue.cc:1035] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4df10e30ed430e910f9cf855fcb4ad" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:25.795876 21896 consensus_queue.cc:1035] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6a3af223df84bfda29789844c01b1bc" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:25.798561 21902 mvcc.cc:205] Tried to move back new op lower bound from 7333857713335451648 to 7333857713302691840. Current Snapshot: MvccSnapshot[applied={T|T < 7333857713335451648}]
I20260927 07:03:25.800544 21411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:25.800624 21411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:25.801306 21411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f378d6c3244b4530a4e5aa2b642e61a9: Sending DeleteTablet for 3 replicas of tablet 30254dd424394dd8b7abc13b6d0262aa
I20260927 07:03:25.801663 21537 tablet_service.cc:1514] Processing DeleteTablet for tablet 30254dd424394dd8b7abc13b6d0262aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:25 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260927 07:03:25.801674 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet 30254dd424394dd8b7abc13b6d0262aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:25 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:57316
I20260927 07:03:25.801748 21805 tablet_service.cc:1514] Processing DeleteTablet for tablet 30254dd424394dd8b7abc13b6d0262aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:25 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:43682
I20260927 07:03:25.801904 21912 tablet_replica.cc:330] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: stopping tablet replica
I20260927 07:03:25.801995 21912 raft_consensus.cc:2229] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:25.802086 21912 raft_consensus.cc:2258] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:25.802188 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21474
I20260927 07:03:25.802986 21912 ts_tablet_manager.cc:1905] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:25.804733 21912 ts_tablet_manager.cc:1918] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:25.804816 21912 log.cc:1196] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/30254dd424394dd8b7abc13b6d0262aa
I20260927 07:03:25.805056 21912 ts_tablet_manager.cc:1939] T 30254dd424394dd8b7abc13b6d0262aa P 2f4df10e30ed430e910f9cf855fcb4ad: Deleting consensus metadata
I20260927 07:03:25.805428 21398 catalog_manager.cc:4841] TS 2f4df10e30ed430e910f9cf855fcb4ad (127.3.57.2:46439): tablet 30254dd424394dd8b7abc13b6d0262aa (table test-table [id=750be1f8d1c34dd18b8546b5e3e83301]) successfully deleted
I20260927 07:03:25.807638 21913 tablet_replica.cc:330] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: stopping tablet replica
I20260927 07:03:25.807747 21913 raft_consensus.cc:2229] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:03:25.807881 21913 raft_consensus.cc:2258] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:03:25.808121 21399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:44083 (error 108)
I20260927 07:03:25.808284 21913 ts_tablet_manager.cc:1905] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:25.808651 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.808548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44083 (local address 127.0.0.1:60330)
0927 07:03:25.808589 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:25.808620 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:44083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 07:03:25.808761 21399 catalog_manager.cc:4584] TS b6a3af223df84bfda29789844c01b1bc (127.3.57.1:44083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30254dd424394dd8b7abc13b6d0262aa: Network error: Client connection negotiation failed: client connection to 127.3.57.1:44083: connect: Connection refused (error 111)
I20260927 07:03:25.809034 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21608
I20260927 07:03:25.810002 21913 ts_tablet_manager.cc:1918] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:25.810081 21913 log.cc:1196] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/30254dd424394dd8b7abc13b6d0262aa
I20260927 07:03:25.810324 21913 ts_tablet_manager.cc:1939] T 30254dd424394dd8b7abc13b6d0262aa P dfc7e9f260024bf2bfcd0e50a1639a87: Deleting consensus metadata
I20260927 07:03:25.810739 21396 catalog_manager.cc:4841] TS dfc7e9f260024bf2bfcd0e50a1639a87 (127.3.57.3:33109): tablet 30254dd424394dd8b7abc13b6d0262aa (table test-table [id=750be1f8d1c34dd18b8546b5e3e83301]) successfully deleted
I20260927 07:03:25.815142 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21742
I20260927 07:03:25.820122 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21380
2026-09-27T07:03:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1896 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T07:03:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:25Z Disabled control of system clock
I20260927 07:03:25.846987 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45439
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:25.924789 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:25.924988 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:25.925009 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:25.926033 21918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:25.926066 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:25.926079 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:25.926090 21918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:25.926102 21918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:25.927318 21918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:45439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:25.927486 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:25.927783 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:25.932404 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.932590 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:25.932631 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:25.932720 21918 server_base.cc:1034] running on GCE node
I20260927 07:03:25.932976 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:25.933243 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:25.934376 21918 hybrid_clock.cc:648] HybridClock initialized: now 1790492605934359 us; error 40 us; skew 500 ppm
I20260927 07:03:25.935490 21918 webserver.cc:460] Webserver started at http://127.3.57.62:44795/ using document root <none> and password file <none>
I20260927 07:03:25.935743 21918 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:25.935789 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:25.935873 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:25.936832 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "e4ec262a3e41814c593e41ae448a3500"
server_key_iv: "203e81ae1f9e408988184d07335ff111"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.937134 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "e4ec262a3e41814c593e41ae448a3500"
server_key_iv: "203e81ae1f9e408988184d07335ff111"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.938287 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:25.938890 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:25.939046 21918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:25.939152 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c"
format_stamp: "Formatted at 2026-09-27 07:03:25 on dist-test-slave-mk3h"
server_key: "e4ec262a3e41814c593e41ae448a3500"
server_key_iv: "203e81ae1f9e408988184d07335ff111"
server_key_version: "encryptionkey@0"
I20260927 07:03:25.939244 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:25.948282 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:25.948576 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:25.948707 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:25.952494 21918 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:45439
I20260927 07:03:25.952522 21984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:45439 every 8 connection(s)
I20260927 07:03:25.952806 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:25.953634 21985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:25.955698 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: Bootstrap starting.
I20260927 07:03:25.956233 21985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:25.956517 21985 log.cc:824] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:25.957110 21985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: No bootstrap required, opened a new log
I20260927 07:03:25.958648 21985 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } }
I20260927 07:03:25.958765 21985 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:25.958786 21985 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a3b55e7de248d0942b7d6bdfbdd34c, State: Initialized, Role: FOLLOWER
I20260927 07:03:25.958941 21985 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } }
I20260927 07:03:25.958998 21985 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:25.959019 21985 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:25.959084 21985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:25.959635 21985 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } }
I20260927 07:03:25.959731 21985 leader_election.cc:302] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5a3b55e7de248d0942b7d6bdfbdd34c; no voters:
I20260927 07:03:25.959879 21985 leader_election.cc:288] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:25.959923 21988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:25.960093 21988 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [term 1 LEADER]: Becoming Leader. State: Replica: c5a3b55e7de248d0942b7d6bdfbdd34c, State: Running, Role: LEADER
I20260927 07:03:25.960106 21985 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:25.960209 21988 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } }
I20260927 07:03:25.960680 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } } }
I20260927 07:03:25.960776 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:25.960788 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 21918
I20260927 07:03:25.960887 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:25.960951 21990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5a3b55e7de248d0942b7d6bdfbdd34c. Latest consensus state: current_term: 1 leader_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a3b55e7de248d0942b7d6bdfbdd34c" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 45439 } } }
I20260927 07:03:25.961004 21990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:25.961071 3300 external_mini_cluster.cc:1428] Setting key cec60c00146bab6673146b846ea01f2a
I20260927 07:03:25.961673 21997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:25.962174 21997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:25.963680 21997 catalog_manager.cc:1347] Generated new cluster ID: 54f4a0480b6744d7a2de92a05d9d1c6e
I20260927 07:03:25.963726 21997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:25.967052 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:25.961477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.3.57.62:45439)
0927 07:03:25.961774 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:25.961783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:25.962014 (+ 231us) server_negotiation.cc:407] Connection header received
0927 07:03:25.962173 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:25.962180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:25.962250 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:25.962380 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:25.962606 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.963852 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.964908 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:25.965474 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:25.965565 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:25.966369 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:25.966372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:25.966372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:25.966548 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:25.966582 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:25.966593 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:25.966640 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:25.966803 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:03:25.966868 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":64,"threads_started":1}
I20260927 07:03:25.984895 21997 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:25.985546 21997 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:25.993396 21997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: Generated new TSK 0
I20260927 07:03:25.993553 21997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:25.994818 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--builtin_ntp_servers=127.3.57.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:26.082989 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:26.083194 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:26.083230 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:26.084343 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:26.084407 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:26.084450 22009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:26.085733 22009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:26.085984 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:26.086274 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:26.091827 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:26.091940 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:26.092026 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:26.092069 22009 server_base.cc:1034] running on GCE node
I20260927 07:03:26.092235 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:26.092473 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:26.093639 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790492606093616 us; error 33 us; skew 500 ppm
I20260927 07:03:26.095072 22009 webserver.cc:460] Webserver started at http://127.3.57.1:45839/ using document root <none> and password file <none>
I20260927 07:03:26.095336 22009 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:26.095391 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:26.095511 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:26.096777 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "54a0ff1979be4231b13a80ba10a9b666"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "331fd6aeeb5eb649329fec26644baf70"
server_key_iv: "cbd528dbd91b432a55b66ae83cce66c8"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.097149 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "54a0ff1979be4231b13a80ba10a9b666"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "331fd6aeeb5eb649329fec26644baf70"
server_key_iv: "cbd528dbd91b432a55b66ae83cce66c8"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.098634 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:03:26.099416 22023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.099618 22009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:26.099720 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "54a0ff1979be4231b13a80ba10a9b666"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "331fd6aeeb5eb649329fec26644baf70"
server_key_iv: "cbd528dbd91b432a55b66ae83cce66c8"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.099792 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:26.114528 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:26.114777 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:26.114919 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:26.115139 22009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:26.115437 22009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:26.115468 22009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.115517 22009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:26.115561 22009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.120834 22009 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:35185
I20260927 07:03:26.120884 22136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:35185 every 8 connection(s)
I20260927 07:03:26.121194 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:26.125728 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.121835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:58749 (local address 127.3.57.62:45439)
0927 07:03:26.121884 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.121886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.122110 (+ 224us) server_negotiation.cc:407] Connection header received
0927 07:03:26.122541 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.122542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.122550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.122603 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:26.123197 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.123727 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.124635 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.124817 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.124865 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.125525 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:26.125527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:26.125527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:26.125530 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:26.125544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:26.125547 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:26.125585 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.125655 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.125678 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:03:26.125880 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.121805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45439 (local address 127.3.57.1:58749)
0927 07:03:26.121989 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.122025 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.122467 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.122650 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.122653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.122660 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:26.123112 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.123116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.123800 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.123801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.124565 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.124567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.124888 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.124889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.124992 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.125415 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:26.125416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:26.125438 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:26.125457 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:26.125596 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:26.125600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:26.125602 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.125646 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.125712 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260927 07:03:26.126375 22137 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45439
I20260927 07:03:26.126478 22137 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:26.126641 22137 heartbeater.cc:511] Master 127.3.57.62:45439 requested a full tablet report, sending...
I20260927 07:03:26.127112 21949 ts_manager.cc:194] Registered new tserver with Master: 54a0ff1979be4231b13a80ba10a9b666 (127.3.57.1:35185)
I20260927 07:03:26.127730 21949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.1:58749
I20260927 07:03:26.130016 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22009
I20260927 07:03:26.130118 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:26.130357 3300 external_mini_cluster.cc:1428] Setting key 1935fc84c1749c6318b5c60c4e61855a
I20260927 07:03:26.131116 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--builtin_ntp_servers=127.3.57.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:26.208530 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:26.208706 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:26.208727 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:26.209764 22140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:26.209796 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:26.209820 22140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:26.211032 22140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:26.211190 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:26.211405 22140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:26.216289 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:26.216280 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:26.216454 22140 server_base.cc:1034] running on GCE node
W20260927 07:03:26.216356 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:26.216707 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:26.216944 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:26.218104 22140 hybrid_clock.cc:648] HybridClock initialized: now 1790492606218086 us; error 30 us; skew 500 ppm
I20260927 07:03:26.219208 22140 webserver.cc:460] Webserver started at http://127.3.57.2:39005/ using document root <none> and password file <none>
I20260927 07:03:26.219439 22140 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:26.219506 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:26.219653 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:26.220633 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "bd14e64511954e0386031861bd59d984"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "343edddd6afae74a74bd487fa5f315b8"
server_key_iv: "01a909c34402c5893571bda94c8a0dad"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.220995 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "bd14e64511954e0386031861bd59d984"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "343edddd6afae74a74bd487fa5f315b8"
server_key_iv: "01a909c34402c5893571bda94c8a0dad"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.222224 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:03:26.222880 22154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.223047 22140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:26.223140 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "bd14e64511954e0386031861bd59d984"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "343edddd6afae74a74bd487fa5f315b8"
server_key_iv: "01a909c34402c5893571bda94c8a0dad"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.223218 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:26.241596 22140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:26.241864 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:26.242007 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:26.242259 22140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:26.242583 22140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:26.242637 22140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.242674 22140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:26.242702 22140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.248250 22140 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:39955
I20260927 07:03:26.248289 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:39955 every 8 connection(s)
I20260927 07:03:26.248606 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:26.253700 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.249348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:46021 (local address 127.3.57.62:45439)
0927 07:03:26.249396 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.249398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.249631 (+ 233us) server_negotiation.cc:407] Connection header received
0927 07:03:26.250131 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.250132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.250148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.250206 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:26.250847 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.251327 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.252274 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.252493 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.252567 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.253403 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:26.253404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:26.253405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:26.253409 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:26.253425 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:26.253436 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:26.253492 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.253582 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.253611 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:03:26.253911 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.249336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45439 (local address 127.3.57.2:46021)
0927 07:03:26.249520 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.249549 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.250028 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.250269 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.250273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.250282 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:26.250756 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.250760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.251422 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.251422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.252176 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.252178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.252572 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.252573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.252731 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.253251 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:26.253251 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:26.253276 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:26.253296 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:26.253494 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:26.253500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:26.253503 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.253584 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.253687 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260927 07:03:26.254340 22268 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45439
I20260927 07:03:26.254412 22268 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:26.254596 22268 heartbeater.cc:511] Master 127.3.57.62:45439 requested a full tablet report, sending...
I20260927 07:03:26.254959 21949 ts_manager.cc:194] Registered new tserver with Master: bd14e64511954e0386031861bd59d984 (127.3.57.2:39955)
I20260927 07:03:26.255307 21949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.2:46021
I20260927 07:03:26.255645 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22140
I20260927 07:03:26.255791 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:26.255983 3300 external_mini_cluster.cc:1428] Setting key 1e14f7f740d0cd605e9762558fd93f92
I20260927 07:03:26.256739 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--builtin_ntp_servers=127.3.57.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:26.335728 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:26.335973 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:26.336006 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:26.337090 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:26.337148 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:26.337189 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:26.338559 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:26.338805 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:26.339097 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:26.344213 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:26.344085 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:26.344188 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:26.344193 22271 server_base.cc:1034] running on GCE node
I20260927 07:03:26.344581 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:26.344811 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:26.345978 22271 hybrid_clock.cc:648] HybridClock initialized: now 1790492606345973 us; error 32 us; skew 500 ppm
I20260927 07:03:26.347054 22271 webserver.cc:460] Webserver started at http://127.3.57.3:42437/ using document root <none> and password file <none>
I20260927 07:03:26.347318 22271 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:26.347388 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:26.347517 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:26.348500 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "badd3883f8914a1b993f11f215dd835b"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "8ab3bb437dc92a70419e6da6e8d5d6f1"
server_key_iv: "fc0b38f371e53111c5a069cbd88c6234"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.348845 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "badd3883f8914a1b993f11f215dd835b"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "8ab3bb437dc92a70419e6da6e8d5d6f1"
server_key_iv: "fc0b38f371e53111c5a069cbd88c6234"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.350081 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:26.350756 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.350922 22271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:26.351024 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "badd3883f8914a1b993f11f215dd835b"
format_stamp: "Formatted at 2026-09-27 07:03:26 on dist-test-slave-mk3h"
server_key: "8ab3bb437dc92a70419e6da6e8d5d6f1"
server_key_iv: "fc0b38f371e53111c5a069cbd88c6234"
server_key_version: "encryptionkey@0"
I20260927 07:03:26.351109 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:26.368815 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:26.369071 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:26.369243 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:26.369452 22271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:26.369748 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:26.369813 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.369853 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:26.369881 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:26.375386 22271 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33065
I20260927 07:03:26.375432 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33065 every 8 connection(s)
I20260927 07:03:26.375730 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:26.380404 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.376429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:47925 (local address 127.3.57.62:45439)
0927 07:03:26.376499 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.376501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.376693 (+ 192us) server_negotiation.cc:407] Connection header received
0927 07:03:26.377138 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.377139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.377149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.377214 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:26.377817 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.378290 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.379249 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.379435 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.379489 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.380185 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:26.380186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:26.380186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:26.380190 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:26.380202 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:26.380206 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:26.380249 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.380319 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.380340 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:03:26.380563 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.376372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:45439 (local address 127.3.57.3:47925)
0927 07:03:26.376542 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.376581 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.377022 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.377266 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.377271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.377279 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:26.377732 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.377735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.378381 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.378381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.379166 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.379168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.379508 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.379508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.379626 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.380067 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:26.380068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:26.380093 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:26.380113 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:26.380255 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:26.380258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:26.380260 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.380307 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.380377 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260927 07:03:26.381062 22399 heartbeater.cc:347] Connected to a master server at 127.3.57.62:45439
I20260927 07:03:26.381104 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22271
I20260927 07:03:26.381168 22399 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:26.381209 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:26.381359 22399 heartbeater.cc:511] Master 127.3.57.62:45439 requested a full tablet report, sending...
I20260927 07:03:26.381397 3300 external_mini_cluster.cc:1428] Setting key a099916957e3005a6bb4478cc2fffcdb
I20260927 07:03:26.381723 21949 ts_manager.cc:194] Registered new tserver with Master: badd3883f8914a1b993f11f215dd835b (127.3.57.3:33065)
I20260927 07:03:26.382151 21949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.3:47925
I20260927 07:03:26.382369 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:26.386224 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.383683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.3.57.62:45439)
0927 07:03:26.383721 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.383723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.383833 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:03:26.383878 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.383879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.383886 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.383933 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:26.384051 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.384456 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.385241 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.385429 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.385488 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.385983 (+ 495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:26.385984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:26.385985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:26.385988 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:26.385998 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:26.386002 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:26.386065 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.386104 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.386112 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:03:26.387820 21949 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:26.389299 21949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:26.395076 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:35185 (local address 127.0.0.1:42178)
0927 07:03:26.391600 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.391646 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.391733 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.391965 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.391966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.391968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:26.392002 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.392004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.392917 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.392918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.394043 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.394046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.394825 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.394874 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.394881 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.394964 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.395014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":93,"threads_started":1}
I20260927 07:03:26.395186 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.3.57.3:33065)
0927 07:03:26.391521 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.391525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.391705 (+ 180us) server_negotiation.cc:407] Connection header received
0927 07:03:26.391828 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.391830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.391843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.391904 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:26.392089 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.393096 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394060 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.394738 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394790 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.394857 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.395083 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.395116 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260927 07:03:26.395377 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33065 (local address 127.0.0.1:52030)
0927 07:03:26.391614 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.391647 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.391733 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.391951 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.391952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.391954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:26.391996 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.391998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.393186 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.393187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.393963 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.393965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.394916 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.394992 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.394995 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.395262 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.395309 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260927 07:03:26.395452 22408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:39955 (local address 127.0.0.1:33490)
0927 07:03:26.391629 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.391655 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.391710 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.391881 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.391883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.391884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:26.391933 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.391935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.393178 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.393178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.394265 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.394266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.395202 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.395203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.395272 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.395275 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.395329 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.395358 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260927 07:03:26.395113 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.3.57.1:35185)
0927 07:03:26.391521 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.391525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.391707 (+ 182us) server_negotiation.cc:407] Connection header received
0927 07:03:26.391821 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.391822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.391836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.391885 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:26.392097 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.392841 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394133 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.394701 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394732 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.394783 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.394999 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.395034 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260927 07:03:26.395679 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.391526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.3.57.2:39955)
0927 07:03:26.391728 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.391732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.391744 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:26.391783 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.391785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.391799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.391872 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:26.391991 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.393092 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.394359 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.395037 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.395094 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.395168 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.395350 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.395401 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260927 07:03:26.396080 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf (DEFAULT_TABLE table=test-table [id=cdead0c1b57d4a28bc4c2988ff002310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:26.396229 22071 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf (DEFAULT_TABLE table=test-table [id=cdead0c1b57d4a28bc4c2988ff002310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:26.396301 22202 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf (DEFAULT_TABLE table=test-table [id=cdead0c1b57d4a28bc4c2988ff002310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:26.396418 22333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:26.396548 22071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:26.396555 22202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:26.398756 22413 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Bootstrap starting.
I20260927 07:03:26.399703 22415 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Bootstrap starting.
I20260927 07:03:26.399794 22413 tablet_bootstrap.cc:654] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:26.400153 22413 log.cc:824] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:26.400359 22415 tablet_bootstrap.cc:654] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:26.400691 22415 log.cc:824] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:26.400918 22413 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: No bootstrap required, opened a new log
I20260927 07:03:26.400996 22413 ts_tablet_manager.cc:1397] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 07:03:26.401022 22414 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Bootstrap starting.
I20260927 07:03:26.401353 22415 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: No bootstrap required, opened a new log
I20260927 07:03:26.401440 22415 ts_tablet_manager.cc:1397] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:26.401625 22414 tablet_bootstrap.cc:654] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:26.401998 22414 log.cc:824] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:26.402931 22414 tablet_bootstrap.cc:492] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: No bootstrap required, opened a new log
I20260927 07:03:26.402998 22414 ts_tablet_manager.cc:1397] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:26.403612 22413 raft_consensus.cc:348] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.403803 22413 raft_consensus.cc:374] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:26.403854 22413 raft_consensus.cc:729] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badd3883f8914a1b993f11f215dd835b, State: Initialized, Role: FOLLOWER
I20260927 07:03:26.403785 22415 raft_consensus.cc:348] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.403960 22415 raft_consensus.cc:374] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:26.404006 22415 raft_consensus.cc:729] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a0ff1979be4231b13a80ba10a9b666, State: Initialized, Role: FOLLOWER
I20260927 07:03:26.403992 22413 consensus_queue.cc:260] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.404114 22415 consensus_queue.cc:260] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.404233 22413 ts_tablet_manager.cc:1428] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260927 07:03:26.404354 22399 heartbeater.cc:503] Master 127.3.57.62:45439 was elected leader, sending a full tablet report...
I20260927 07:03:26.404351 22415 ts_tablet_manager.cc:1428] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:26.404603 22137 heartbeater.cc:503] Master 127.3.57.62:45439 was elected leader, sending a full tablet report...
I20260927 07:03:26.405565 22414 raft_consensus.cc:348] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.405673 22414 raft_consensus.cc:374] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:26.405692 22414 raft_consensus.cc:729] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd14e64511954e0386031861bd59d984, State: Initialized, Role: FOLLOWER
I20260927 07:03:26.405793 22414 consensus_queue.cc:260] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.405997 22414 ts_tablet_manager.cc:1428] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:26.406149 22268 heartbeater.cc:503] Master 127.3.57.62:45439 was elected leader, sending a full tablet report...
W20260927 07:03:26.499241 22269 tablet.cc:2369] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:03:26.592514 22421 raft_consensus.cc:482] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:26.592670 22421 raft_consensus.cc:504] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.593346 22421 leader_election.cc:288] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54a0ff1979be4231b13a80ba10a9b666 (127.3.57.1:35185), badd3883f8914a1b993f11f215dd835b (127.3.57.3:33065)
I20260927 07:03:26.596979 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.593723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:48807 (local address 127.3.57.3:33065)
0927 07:03:26.593768 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.593770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.593836 (+ 66us) server_negotiation.cc:407] Connection header received
0927 07:03:26.593936 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.593936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.593947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.593992 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:26.594126 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.594720 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.596289 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.596647 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.596753 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.596763 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.596880 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.596912 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 07:03:26.596990 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.593648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33065 (local address 127.3.57.2:48807)
0927 07:03:26.593783 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.593793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.593847 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.594003 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.594003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.594005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:26.594073 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.594075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.594799 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.594800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.596161 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.596164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.596743 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.596744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.596818 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.596823 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.596875 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.596911 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":44,"thread_start_us":77,"threads_started":1}
I20260927 07:03:26.597337 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.593273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:35185 (local address 127.3.57.2:56065)
0927 07:03:26.593316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.593342 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:26.593397 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:26.593733 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:26.593734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:26.593736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:26.593808 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.593810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.594862 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.594863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.596177 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:26.596180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.597063 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.597064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:26.597146 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.597150 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:26.597198 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 07:03:26.597245 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 07:03:26.597467 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2fa9d811e5402cb7e559d6c0cf6adf" candidate_uuid: "bd14e64511954e0386031861bd59d984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badd3883f8914a1b993f11f215dd835b" is_pre_election: true
I20260927 07:03:26.597632 22353 raft_consensus.cc:2454] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd14e64511954e0386031861bd59d984 in term 0.
I20260927 07:03:26.597841 22158 leader_election.cc:302] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: badd3883f8914a1b993f11f215dd835b, bd14e64511954e0386031861bd59d984; no voters:
I20260927 07:03:26.597899 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.593495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:56065 (local address 127.3.57.1:35185)
0927 07:03:26.593551 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.593554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.593566 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:26.593598 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.593599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.593613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.593676 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:26.593899 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.594757 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.596285 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.596943 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.597015 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.597026 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.597761 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.597813 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 07:03:26.598080 22421 raft_consensus.cc:2781] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:26.598138 22421 raft_consensus.cc:482] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:26.598169 22421 raft_consensus.cc:3037] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:26.598218 22091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2fa9d811e5402cb7e559d6c0cf6adf" candidate_uuid: "bd14e64511954e0386031861bd59d984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a0ff1979be4231b13a80ba10a9b666" is_pre_election: true
I20260927 07:03:26.598376 22091 raft_consensus.cc:2454] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd14e64511954e0386031861bd59d984 in term 0.
I20260927 07:03:26.598981 22421 raft_consensus.cc:504] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.599228 22353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2fa9d811e5402cb7e559d6c0cf6adf" candidate_uuid: "bd14e64511954e0386031861bd59d984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badd3883f8914a1b993f11f215dd835b"
I20260927 07:03:26.599282 22091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2fa9d811e5402cb7e559d6c0cf6adf" candidate_uuid: "bd14e64511954e0386031861bd59d984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a0ff1979be4231b13a80ba10a9b666"
I20260927 07:03:26.599293 22353 raft_consensus.cc:3037] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:26.599368 22091 raft_consensus.cc:3037] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:26.600184 22353 raft_consensus.cc:2454] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd14e64511954e0386031861bd59d984 in term 1.
I20260927 07:03:26.599177 22421 leader_election.cc:288] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [CANDIDATE]: Term 1 election: Requested vote from peers 54a0ff1979be4231b13a80ba10a9b666 (127.3.57.1:35185), badd3883f8914a1b993f11f215dd835b (127.3.57.3:33065)
I20260927 07:03:26.600283 22091 raft_consensus.cc:2454] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd14e64511954e0386031861bd59d984 in term 1.
I20260927 07:03:26.600340 22158 leader_election.cc:302] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: badd3883f8914a1b993f11f215dd835b, bd14e64511954e0386031861bd59d984; no voters:
I20260927 07:03:26.600401 22421 raft_consensus.cc:2781] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:26.600507 22421 raft_consensus.cc:686] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 1 LEADER]: Becoming Leader. State: Replica: bd14e64511954e0386031861bd59d984, State: Running, Role: LEADER
I20260927 07:03:26.600586 22421 consensus_queue.cc:237] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } }
I20260927 07:03:26.601403 21948 catalog_manager.cc:5495] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd14e64511954e0386031861bd59d984 (127.3.57.2). New cstate: current_term: 1 leader_uuid: "bd14e64511954e0386031861bd59d984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd14e64511954e0386031861bd59d984" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 39955 } health_report { overall_health: HEALTHY } } }
I20260927 07:03:26.619160 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.617039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.3.57.2:39955)
0927 07:03:26.617098 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:26.617100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:26.617107 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:03:26.617140 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:26.617141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:26.617161 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:26.617219 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:26.617376 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.617798 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.618488 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:26.618650 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:26.618694 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:26.618830 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:26.618909 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:26.618975 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:26.619034 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 07:03:26.619061 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:03:26.620841 22353 raft_consensus.cc:1261] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 1 FOLLOWER]: Refusing update from remote peer bd14e64511954e0386031861bd59d984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:26.620868 22091 raft_consensus.cc:1261] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666 [term 1 FOLLOWER]: Refusing update from remote peer bd14e64511954e0386031861bd59d984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:26.621188 22423 consensus_queue.cc:1035] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badd3883f8914a1b993f11f215dd835b" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:26.621306 22421 consensus_queue.cc:1035] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a0ff1979be4231b13a80ba10a9b666" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 07:03:26.621804 22138 tablet.cc:2369] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:03:26.623252 22426 mvcc.cc:205] Tried to move back new op lower bound from 7333857716716462080 to 7333857716636536832. Current Snapshot: MvccSnapshot[applied={T|T < 7333857716716462080}]
I20260927 07:03:26.623791 22428 mvcc.cc:205] Tried to move back new op lower bound from 7333857716716462080 to 7333857716636536832. Current Snapshot: MvccSnapshot[applied={T|T < 7333857716716462080}]
I20260927 07:03:26.623822 22427 mvcc.cc:205] Tried to move back new op lower bound from 7333857716716462080 to 7333857716636536832. Current Snapshot: MvccSnapshot[applied={T|T < 7333857716716462080}]
I20260927 07:03:26.626236 21948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:26.626349 21948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:26.627035 21948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5a3b55e7de248d0942b7d6bdfbdd34c: Sending DeleteTablet for 3 replicas of tablet 1e2fa9d811e5402cb7e559d6c0cf6adf
I20260927 07:03:26.627378 22071 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:26 UTC) from {username='slave'} at 127.0.0.1:42178
I20260927 07:03:26.627410 22202 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:26 UTC) from {username='slave'} at 127.0.0.1:33490
I20260927 07:03:26.627441 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:26 UTC) from {username='slave'} at 127.0.0.1:52030
I20260927 07:03:26.627609 22435 tablet_replica.cc:330] T 1e2fa9d811e5402cb7e559d6c0cf6adf P 54a0ff1979be4231b13a80ba10a9b666: stopping tablet replica
I20260927 07:03:26.627707 22436 tablet_replica.cc:330] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: stopping tablet replica
I20260927 07:03:26.627801 22436 raft_consensus.cc:2229] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:03:26.627815 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22009
I20260927 07:03:26.628424 22436 raft_consensus.cc:2258] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:26.628825 22436 ts_tablet_manager.cc:1905] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:26.630410 22436 ts_tablet_manager.cc:1918] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:26.630479 22436 log.cc:1196] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-1/wal/wals/1e2fa9d811e5402cb7e559d6c0cf6adf
I20260927 07:03:26.630715 22436 ts_tablet_manager.cc:1939] T 1e2fa9d811e5402cb7e559d6c0cf6adf P bd14e64511954e0386031861bd59d984: Deleting consensus metadata
I20260927 07:03:26.631059 21936 catalog_manager.cc:4841] TS bd14e64511954e0386031861bd59d984 (127.3.57.2:39955): tablet 1e2fa9d811e5402cb7e559d6c0cf6adf (table test-table [id=cdead0c1b57d4a28bc4c2988ff002310]) successfully deleted
W20260927 07:03:26.633024 21935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:35185 (error 108)
I20260927 07:03:26.633894 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22140
I20260927 07:03:26.634487 22437 tablet_replica.cc:330] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: stopping tablet replica
I20260927 07:03:26.634547 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:26.634012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:35185 (local address 127.0.0.1:42184)
0927 07:03:26.634458 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:26.634497 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:35185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260927 07:03:26.634620 22437 raft_consensus.cc:2229] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:26.634714 22437 raft_consensus.cc:2258] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 07:03:26.634725 21935 catalog_manager.cc:4584] TS 54a0ff1979be4231b13a80ba10a9b666 (127.3.57.1:35185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e2fa9d811e5402cb7e559d6c0cf6adf: Network error: Client connection negotiation failed: client connection to 127.3.57.1:35185: connect: Connection refused (error 111)
I20260927 07:03:26.635124 22437 ts_tablet_manager.cc:1905] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:26.636951 22437 ts_tablet_manager.cc:1918] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:26.637007 22437 log.cc:1196] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/1e2fa9d811e5402cb7e559d6c0cf6adf
I20260927 07:03:26.637236 22437 ts_tablet_manager.cc:1939] T 1e2fa9d811e5402cb7e559d6c0cf6adf P badd3883f8914a1b993f11f215dd835b: Deleting consensus metadata
I20260927 07:03:26.637741 21936 catalog_manager.cc:4841] TS badd3883f8914a1b993f11f215dd835b (127.3.57.3:33065): tablet 1e2fa9d811e5402cb7e559d6c0cf6adf (table test-table [id=cdead0c1b57d4a28bc4c2988ff002310]) successfully deleted
I20260927 07:03:26.639238 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22271
I20260927 07:03:26.644570 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 21918
2026-09-27T07:03:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (823 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:26 dist-test-slave-mk3h krb5kdc[22441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:26 dist-test-slave-mk3h krb5kdc[22441](info): set up 2 sockets
Sep 27 07:03:26 dist-test-slave-mk3h krb5kdc[22441](info): commencing operation
krb5kdc: starting...
W20260927 07:03:27.659400 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:27 dist-test-slave-mk3h krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:27.778424 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:39587
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:39587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:27.855603 22454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:27.855818 22454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:27.855851 22454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:27.856956 22454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:27.857023 22454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:27.857048 22454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:27.857074 22454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:27.857105 22454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:27.858546 22454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:39587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:39587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:27.858812 22454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:27.859045 22454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:27.863893 22459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:27.863965 22460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:27.863945 22454 server_base.cc:1034] running on GCE node
W20260927 07:03:27.864127 22462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:27.864398 22454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:27.864718 22454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:27.865869 22454 hybrid_clock.cc:648] HybridClock initialized: now 1790492607865851 us; error 38 us; skew 500 ppm
Sep 27 07:03:27 dist-test-slave-mk3h krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:27.879052 22454 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:27.879459 22454 webserver.cc:460] Webserver started at http://127.3.57.62:43701/ using document root <none> and password file <none>
I20260927 07:03:27.879719 22454 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:27.879789 22454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:27.879940 22454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:27.880970 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "3ac2fb4484b34e8db9e64ae085e8e944"
format_stamp: "Formatted at 2026-09-27 07:03:27 on dist-test-slave-mk3h"
server_key: "ed313cf9dd9c83d301b3db3f6d622c69"
server_key_iv: "8f367a3ee3a6ddafbc8915c3d6e9dd51"
server_key_version: "encryptionkey@0"
I20260927 07:03:27.881313 22454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "3ac2fb4484b34e8db9e64ae085e8e944"
format_stamp: "Formatted at 2026-09-27 07:03:27 on dist-test-slave-mk3h"
server_key: "ed313cf9dd9c83d301b3db3f6d622c69"
server_key_iv: "8f367a3ee3a6ddafbc8915c3d6e9dd51"
server_key_version: "encryptionkey@0"
I20260927 07:03:27.882618 22454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:27.883304 22469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:27.883481 22454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:27.883591 22454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "3ac2fb4484b34e8db9e64ae085e8e944"
format_stamp: "Formatted at 2026-09-27 07:03:27 on dist-test-slave-mk3h"
server_key: "ed313cf9dd9c83d301b3db3f6d622c69"
server_key_iv: "8f367a3ee3a6ddafbc8915c3d6e9dd51"
server_key_version: "encryptionkey@0"
I20260927 07:03:27.883667 22454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:27.905227 22454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:27.906903 22454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:27.907054 22454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:27.911106 22454 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:39587
I20260927 07:03:27.911149 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:39587 every 8 connection(s)
I20260927 07:03:27.911510 22454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:27.912350 22522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:27.912822 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22454
I20260927 07:03:27.912932 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:27.913123 3300 external_mini_cluster.cc:1428] Setting key c71b16d3f7b6a9f92b99f11547480643
I20260927 07:03:27.915298 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: Bootstrap starting.
I20260927 07:03:27.915934 22522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:27.916262 22522 log.cc:824] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:27.916960 22522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: No bootstrap required, opened a new log
I20260927 07:03:27.919404 22522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } }
I20260927 07:03:27.919521 22522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:27.919585 22522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac2fb4484b34e8db9e64ae085e8e944, State: Initialized, Role: FOLLOWER
I20260927 07:03:27.919720 22522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } }
I20260927 07:03:27.919806 22522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:27.919848 22522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 27 07:03:27 dist-test-slave-mk3h krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:27.919893 22522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:27.920373 22522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } }
I20260927 07:03:27.920482 22522 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac2fb4484b34e8db9e64ae085e8e944; no voters:
I20260927 07:03:27.920629 22522 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:27.920706 22527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:27.920874 22522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:27.921170 22527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac2fb4484b34e8db9e64ae085e8e944, State: Running, Role: LEADER
I20260927 07:03:27.921284 22527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } }
I20260927 07:03:27.922597 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } } }
I20260927 07:03:27.922688 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:27.922883 22529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac2fb4484b34e8db9e64ae085e8e944. Latest consensus state: current_term: 1 leader_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac2fb4484b34e8db9e64ae085e8e944" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 39587 } } }
I20260927 07:03:27.922960 22529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:27.923125 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:27.923879 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:27.924045 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:27.913653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.3.57.62:39587)
0927 07:03:27.913809 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:27.913817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:27.913849 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:03:27.914221 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:27.914226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:27.914266 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:27.914378 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:27.915146 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:27.916198 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:27.917245 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:27.917636 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:27.917716 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:27.920306 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:27.920310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:27.920311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:27.920322 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:27.922204 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:27.922593 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:27.922596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:27.922596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:27.922738 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:27.922954 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:27.922955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:27.922956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:27.923260 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:27.923412 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:27.923689 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:03:27.923841 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260927 07:03:27.925457 22542 catalog_manager.cc:1347] Generated new cluster ID: c29ddf6f8ea94121bcec0f05cf792a8e
I20260927 07:03:27.925506 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:27.932499 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:27.932962 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:27.936615 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: Generated new TSK 0
I20260927 07:03:27.936779 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.57.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
Entry for principal kudu/127.3.57.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab.
I20260927 07:03:27.963745 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:28.041508 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.041721 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.041757 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.042884 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.042948 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.042994 22548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:28.044466 22548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.1.keytab
--principal=kudu/127.3.57.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.044710 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.044934 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.049755 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.049784 22548 server_base.cc:1034] running on GCE node
W20260927 07:03:28.049757 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.049774 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.050186 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.050417 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.051589 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790492608051569 us; error 39 us; skew 500 ppm
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:28.053999 22548 init.cc:377] Logged in from keytab as kudu/127.3.57.1@KRBTEST.COM (short username kudu)
I20260927 07:03:28.054391 22548 webserver.cc:460] Webserver started at http://127.3.57.1:46505/ using document root <none> and password file <none>
I20260927 07:03:28.054593 22548 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.054634 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.054766 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.055790 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "6f3bef6edacb4ea69409638826470d11"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "3939091ed15daac6b426cc322e41542b"
server_key_iv: "8359ef6f617d5c9133f76339d0e0f731"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.056133 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "6f3bef6edacb4ea69409638826470d11"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "3939091ed15daac6b426cc322e41542b"
server_key_iv: "8359ef6f617d5c9133f76339d0e0f731"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.057348 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:28.058041 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.058208 22548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:28.058290 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "6f3bef6edacb4ea69409638826470d11"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "3939091ed15daac6b426cc322e41542b"
server_key_iv: "8359ef6f617d5c9133f76339d0e0f731"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.058358 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.068235 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.070379 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.070509 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.070704 22548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:28.070983 22548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:28.071015 22548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.071034 22548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:28.071080 22548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.076671 22548 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:44659
I20260927 07:03:28.076745 22676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:44659 every 8 connection(s)
I20260927 07:03:28.077028 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:28.077993 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22548
I20260927 07:03:28.078110 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:28.078307 3300 external_mini_cluster.cc:1428] Setting key 13132334fb7780ec9e0ce618046b7e01
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.1@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.2@KRBTEST.COM; defaulting to no policy
I20260927 07:03:28.087100 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.077678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:37605 (local address 127.3.57.62:39587)
0927 07:03:28.077728 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.077730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.077941 (+ 211us) server_negotiation.cc:407] Connection header received
0927 07:03:28.078581 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.078583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.078594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.078660 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.079672 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.080469 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.081454 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.081676 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.081764 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.084187 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.084188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.084188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:28.084194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.085641 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.086137 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.086138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.086139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.086204 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.086472 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.086473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.086474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.086577 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.086630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.086850 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.086962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:03:28.087172 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.077657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:39587 (local address 127.3.57.1:37605)
0927 07:03:28.077798 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.077846 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.078441 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.078720 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.078724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.079353 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:28.079542 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.079546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.080565 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.080566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.081347 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.081349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.081774 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.081775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.081902 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.082397 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:28.082405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:28.084073 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:28.085725 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.085732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.085734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.086042 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.086284 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.086285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.086285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.086407 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.086637 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:28.086642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:28.086745 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.086822 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.086992 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260927 07:03:28.087699 22677 heartbeater.cc:347] Connected to a master server at 127.3.57.62:39587
I20260927 07:03:28.087860 22677 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:28.088047 22677 heartbeater.cc:511] Master 127.3.57.62:39587 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.2@KRBTEST.COM" created.
I20260927 07:03:28.088621 22486 ts_manager.cc:194] Registered new tserver with Master: 6f3bef6edacb4ea69409638826470d11 (127.3.57.1:44659)
I20260927 07:03:28.089241 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.1@KRBTEST.COM'} at 127.3.57.1:37605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
Entry for principal kudu/127.3.57.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab.
I20260927 07:03:28.101595 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:28.180284 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.180466 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.180488 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.181607 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.181648 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.181702 22682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:28.183105 22682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.2.keytab
--principal=kudu/127.3.57.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.183329 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.183540 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.188450 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.188490 22682 server_base.cc:1034] running on GCE node
W20260927 07:03:28.188496 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.188455 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.188817 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.189039 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.190196 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790492608190181 us; error 43 us; skew 500 ppm
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:28.192600 22682 init.cc:377] Logged in from keytab as kudu/127.3.57.2@KRBTEST.COM (short username kudu)
I20260927 07:03:28.193038 22682 webserver.cc:460] Webserver started at http://127.3.57.2:44387/ using document root <none> and password file <none>
I20260927 07:03:28.193255 22682 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.193320 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.193473 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.194433 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "0ffe1093d08f4375a244adc0f3eec3c6"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "5df827d270a466f39d4f80f5081fb0fb"
server_key_iv: "9267e73d815bdfd24a32c53e1804e160"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.194778 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "0ffe1093d08f4375a244adc0f3eec3c6"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "5df827d270a466f39d4f80f5081fb0fb"
server_key_iv: "9267e73d815bdfd24a32c53e1804e160"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.196152 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:28.196877 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.197063 22682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:28.197160 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "0ffe1093d08f4375a244adc0f3eec3c6"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "5df827d270a466f39d4f80f5081fb0fb"
server_key_iv: "9267e73d815bdfd24a32c53e1804e160"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.197235 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.212707 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.214519 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.214675 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.214890 22682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:28.215220 22682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:28.215278 22682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.215312 22682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:28.215340 22682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.220885 22682 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:44381
I20260927 07:03:28.220932 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:44381 every 8 connection(s)
I20260927 07:03:28.221289 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:28.225672 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22682
I20260927 07:03:28.225787 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:28.225980 3300 external_mini_cluster.cc:1428] Setting key 77d20df85a8e4cd9b765aadf22359ad1
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.2@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:28.231220 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.221975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:39587 (local address 127.3.57.2:34467)
0927 07:03:28.222106 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.222150 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.222738 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.223002 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.223005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.223601 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:28.223796 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.223800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.224505 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.224506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.225293 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.225295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.225716 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.225716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.225894 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.226451 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:28.226460 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:28.228114 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:28.229939 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.229945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.229946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.230155 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.230374 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.230374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.230375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.230466 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.230763 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:28.230768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:28.230883 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.230964 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.231075 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260927 07:03:28.231213 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.222003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:34467 (local address 127.3.57.62:39587)
0927 07:03:28.222047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.222050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.222249 (+ 199us) server_negotiation.cc:407] Connection header received
0927 07:03:28.222860 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.222861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.222875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.222926 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.223887 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.224404 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.225386 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.225620 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.225706 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.228248 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.228250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.228250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:28.228259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.229859 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.230237 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.230238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.230239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.230303 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.230537 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.230538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.230538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.230692 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.230770 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.231013 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.231102 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:03:28.231729 22811 heartbeater.cc:347] Connected to a master server at 127.3.57.62:39587
I20260927 07:03:28.231830 22811 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:28.232024 22811 heartbeater.cc:511] Master 127.3.57.62:39587 requested a full tablet report, sending...
I20260927 07:03:28.232426 22486 ts_manager.cc:194] Registered new tserver with Master: 0ffe1093d08f4375a244adc0f3eec3c6 (127.3.57.2:44381)
I20260927 07:03:28.232792 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.2@KRBTEST.COM'} at 127.3.57.2:34467
WARNING: no policy specified for kudu/127.3.57.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
Entry for principal kudu/127.3.57.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab.
I20260927 07:03:28.251006 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:28.328716 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.328919 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.328954 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.330086 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.330150 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.330195 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:28.331676 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.3.keytab
--principal=kudu/127.3.57.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:39587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.331933 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.332142 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.336930 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.336858 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.336983 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.337059 22816 server_base.cc:1034] running on GCE node
I20260927 07:03:28.337242 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.337472 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.338629 22816 hybrid_clock.cc:648] HybridClock initialized: now 1790492608338616 us; error 24 us; skew 500 ppm
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:28.341152 22816 init.cc:377] Logged in from keytab as kudu/127.3.57.3@KRBTEST.COM (short username kudu)
I20260927 07:03:28.341575 22816 webserver.cc:460] Webserver started at http://127.3.57.3:34839/ using document root <none> and password file <none>
I20260927 07:03:28.341802 22816 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.341869 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.342008 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.342947 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "96a4b49251da450988f5244bfb21462c"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "ca6155c1d6a805396a2b04c3673ead68"
server_key_iv: "ec99aff0b1c5fc2003cbf8c637fcb9d9"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.343276 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "96a4b49251da450988f5244bfb21462c"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "ca6155c1d6a805396a2b04c3673ead68"
server_key_iv: "ec99aff0b1c5fc2003cbf8c637fcb9d9"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.344563 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:28.345347 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.345538 22816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:28.345629 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "96a4b49251da450988f5244bfb21462c"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "ca6155c1d6a805396a2b04c3673ead68"
server_key_iv: "ec99aff0b1c5fc2003cbf8c637fcb9d9"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.345711 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.354116 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.356455 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.356599 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.356799 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:28.357069 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:28.357120 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.357156 22816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:28.357182 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.363278 22816 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:46211
I20260927 07:03:28.363328 22944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:46211 every 8 connection(s)
I20260927 07:03:28.363662 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:28.365156 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22816
I20260927 07:03:28.365324 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:28.365495 3300 external_mini_cluster.cc:1428] Setting key e04b7febfc822f1340012ee94d148742
Sep 27 07:03:28 dist-test-slave-mk3h krb5kdc[22441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.3@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:28.372650 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.364338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:55199 (local address 127.3.57.62:39587)
0927 07:03:28.364397 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.364399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.364588 (+ 189us) server_negotiation.cc:407] Connection header received
0927 07:03:28.365016 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.365017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.365031 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.365114 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.366122 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.366633 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.367703 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.367920 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.368005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.370054 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.370055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.370056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:28.370064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.371257 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.371703 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.371704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.371704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.371776 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.372023 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.372024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.372024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.372142 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.372217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.372482 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.372560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 07:03:28.372820 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.364309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:39587 (local address 127.3.57.3:55199)
0927 07:03:28.364469 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.364507 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.364929 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.365157 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.365160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.365790 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:28.366027 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.366032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.366726 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.366728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.367513 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.367516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.368015 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.368016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.368145 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.368624 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:28.368632 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:28.369958 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:28.371339 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.371345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.371357 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.371627 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.371842 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:28.371843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:28.371843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:28.371952 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:28.372208 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:28.372214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:28.372354 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.372460 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.372608 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 07:03:28.373314 22945 heartbeater.cc:347] Connected to a master server at 127.3.57.62:39587
I20260927 07:03:28.373402 22945 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:28.373525 22945 heartbeater.cc:511] Master 127.3.57.62:39587 requested a full tablet report, sending...
I20260927 07:03:28.373883 22486 ts_manager.cc:194] Registered new tserver with Master: 96a4b49251da450988f5244bfb21462c (127.3.57.3:46211)
I20260927 07:03:28.374253 22486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.57.3@KRBTEST.COM'} at 127.3.57.3:55199
I20260927 07:03:28.377202 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:28.384055 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.378483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.3.57.62:39587)
0927 07:03:28.378542 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.378543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.378637 (+ 94us) server_negotiation.cc:407] Connection header received
0927 07:03:28.378716 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.378716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.378724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.378769 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.379266 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.379755 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.380432 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.380599 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.380647 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.381746 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.381747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.381747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:28.381753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.382998 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.383372 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.383373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.383374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.383425 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:28.383619 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:28.383620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:28.383620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:28.383746 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.383823 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.383938 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.384000 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 07:03:28.385673 22486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:28.387338 22486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:28.393329 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.3.57.2:44381)
0927 07:03:28.389873 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.389878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.389888 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:28.389922 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.389923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.389937 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.390218 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:28.390315 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.390970 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.392189 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.392911 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.392993 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.393051 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.393203 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.393248 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260927 07:03:28.393368 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:44381 (local address 127.0.0.1:59322)
0927 07:03:28.389669 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.389678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.389720 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.390093 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.390094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.390097 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:28.390147 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.390148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.391080 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.391081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.392100 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.392102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.393015 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.393016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.393090 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.393103 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.393180 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.393235 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":71,"threads_started":1}
I20260927 07:03:28.393492 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44659 (local address 127.0.0.1:43890)
0927 07:03:28.389475 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.389510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.389566 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.389752 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.389754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.389757 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:28.389816 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.389818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.391205 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.391206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.392355 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.392358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.393287 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.393287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.393343 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.393346 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.393395 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.393436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260927 07:03:28.393741 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.3.57.1:44659)
0927 07:03:28.389565 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.389571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.389583 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:28.389634 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.389636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.389653 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.389738 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:28.389885 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.391096 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.392443 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.393147 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.393205 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.393254 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.393611 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.393659 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260927 07:03:28.394255 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.3.57.3:46211)
0927 07:03:28.390075 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.390079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.390187 (+ 108us) server_negotiation.cc:407] Connection header received
0927 07:03:28.390297 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.390299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.390312 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.390370 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:28.390552 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.391582 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.392892 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.393827 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.393891 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.393937 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.394135 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.394181 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260927 07:03:28.394287 22745 tablet_service.cc:1467] Processing CreateTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae (DEFAULT_TABLE table=test-table [id=4db91709863d425f96fa50aea3c425a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:28.394457 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.389537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:46211 (local address 127.0.0.1:42526)
0927 07:03:28.390125 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.390135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.390215 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.390452 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.390453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.390455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:28.390490 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.390491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.391685 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.391687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.392738 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.392740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.393973 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.393974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.394038 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.394042 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.394099 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.394134 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":80,"threads_started":1}
I20260927 07:03:28.394371 22611 tablet_service.cc:1467] Processing CreateTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae (DEFAULT_TABLE table=test-table [id=4db91709863d425f96fa50aea3c425a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:28.394651 22745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c4921e3fc04a3686365dc7b4fa22ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:28.394693 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c4921e3fc04a3686365dc7b4fa22ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:28.395081 22879 tablet_service.cc:1467] Processing CreateTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae (DEFAULT_TABLE table=test-table [id=4db91709863d425f96fa50aea3c425a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:28.395388 22879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c4921e3fc04a3686365dc7b4fa22ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:28.397281 22959 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Bootstrap starting.
I20260927 07:03:28.397871 22959 tablet_bootstrap.cc:654] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:28.398202 22960 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: Bootstrap starting.
I20260927 07:03:28.398487 22959 log.cc:824] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:28.398655 22961 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Bootstrap starting.
I20260927 07:03:28.399065 22960 tablet_bootstrap.cc:654] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:28.399318 22961 tablet_bootstrap.cc:654] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:28.399595 22959 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: No bootstrap required, opened a new log
I20260927 07:03:28.399668 22959 ts_tablet_manager.cc:1397] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:28.399752 22960 log.cc:824] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:28.399976 22961 log.cc:824] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:28.400472 22960 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: No bootstrap required, opened a new log
I20260927 07:03:28.400580 22960 ts_tablet_manager.cc:1397] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:03:28.400754 22961 tablet_bootstrap.cc:492] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: No bootstrap required, opened a new log
I20260927 07:03:28.400813 22961 ts_tablet_manager.cc:1397] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:03:28.401466 22959 raft_consensus.cc:348] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.401615 22959 raft_consensus.cc:374] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:28.401656 22959 raft_consensus.cc:729] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffe1093d08f4375a244adc0f3eec3c6, State: Initialized, Role: FOLLOWER
I20260927 07:03:28.401774 22959 consensus_queue.cc:260] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.402050 22811 heartbeater.cc:503] Master 127.3.57.62:39587 was elected leader, sending a full tablet report...
I20260927 07:03:28.402329 22959 ts_tablet_manager.cc:1428] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:03:28.402498 22960 raft_consensus.cc:348] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.402570 22961 raft_consensus.cc:348] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.402670 22960 raft_consensus.cc:374] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:28.402705 22961 raft_consensus.cc:374] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:28.402724 22960 raft_consensus.cc:729] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3bef6edacb4ea69409638826470d11, State: Initialized, Role: FOLLOWER
I20260927 07:03:28.402738 22961 raft_consensus.cc:729] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a4b49251da450988f5244bfb21462c, State: Initialized, Role: FOLLOWER
I20260927 07:03:28.402828 22960 consensus_queue.cc:260] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.402845 22961 consensus_queue.cc:260] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.403086 22961 ts_tablet_manager.cc:1428] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:03:28.403086 22960 ts_tablet_manager.cc:1428] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 07:03:28.403200 22677 heartbeater.cc:503] Master 127.3.57.62:39587 was elected leader, sending a full tablet report...
I20260927 07:03:28.403435 22945 heartbeater.cc:503] Master 127.3.57.62:39587 was elected leader, sending a full tablet report...
W20260927 07:03:28.471885 22812 tablet.cc:2369] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:03:28.490338 22967 raft_consensus.cc:482] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:28.490449 22967 raft_consensus.cc:504] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.490855 22967 leader_election.cc:288] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3bef6edacb4ea69409638826470d11 (127.3.57.1:44659), 0ffe1093d08f4375a244adc0f3eec3c6 (127.3.57.2:44381)
I20260927 07:03:28.493783 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.490991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44659 (local address 127.3.57.3:38005)
0927 07:03:28.491060 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.491074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.491119 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.491339 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.491340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.491342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:28.491417 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.491419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.492147 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.492148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.492993 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.492996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.493539 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.493617 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.493619 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.493661 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.493690 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":25}
I20260927 07:03:28.493808 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.491117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:38005 (local address 127.3.57.1:44659)
0927 07:03:28.491160 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.491163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.491171 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:28.491201 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.491202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.491215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.491280 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:28.491494 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.492070 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493087 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.493456 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493510 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.493540 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.493688 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.493736 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:03:28.493850 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.491082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:52459 (local address 127.3.57.2:44381)
0927 07:03:28.491130 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.491132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.491277 (+ 145us) server_negotiation.cc:407] Connection header received
0927 07:03:28.491357 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.491359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.491371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.491431 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:28.491661 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.492126 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493081 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.493462 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493549 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.493558 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.493738 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.493776 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 07:03:28.493875 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.491016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:44381 (local address 127.3.57.3:52459)
0927 07:03:28.491203 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.491213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.491269 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.491445 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.491445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.491447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:28.491545 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.491546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.492206 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.492207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.492991 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.492993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.493571 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.493572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.493642 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.493664 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.493722 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.493756 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260927 07:03:28.494140 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c4921e3fc04a3686365dc7b4fa22ae" candidate_uuid: "96a4b49251da450988f5244bfb21462c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3bef6edacb4ea69409638826470d11" is_pre_election: true
I20260927 07:03:28.494253 22631 raft_consensus.cc:2454] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a4b49251da450988f5244bfb21462c in term 0.
I20260927 07:03:28.494501 22834 leader_election.cc:302] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3bef6edacb4ea69409638826470d11, 96a4b49251da450988f5244bfb21462c; no voters:
I20260927 07:03:28.494545 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c4921e3fc04a3686365dc7b4fa22ae" candidate_uuid: "96a4b49251da450988f5244bfb21462c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" is_pre_election: true
I20260927 07:03:28.494683 22765 raft_consensus.cc:2454] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a4b49251da450988f5244bfb21462c in term 0.
I20260927 07:03:28.494737 22967 raft_consensus.cc:2781] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:28.494807 22967 raft_consensus.cc:482] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:28.494838 22967 raft_consensus.cc:3037] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:28.495532 22967 raft_consensus.cc:504] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.495683 22967 leader_election.cc:288] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [CANDIDATE]: Term 1 election: Requested vote from peers 6f3bef6edacb4ea69409638826470d11 (127.3.57.1:44659), 0ffe1093d08f4375a244adc0f3eec3c6 (127.3.57.2:44381)
I20260927 07:03:28.495841 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c4921e3fc04a3686365dc7b4fa22ae" candidate_uuid: "96a4b49251da450988f5244bfb21462c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3bef6edacb4ea69409638826470d11"
I20260927 07:03:28.495834 22765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c4921e3fc04a3686365dc7b4fa22ae" candidate_uuid: "96a4b49251da450988f5244bfb21462c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ffe1093d08f4375a244adc0f3eec3c6"
I20260927 07:03:28.495923 22765 raft_consensus.cc:3037] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:28.495927 22631 raft_consensus.cc:3037] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:28.496623 22765 raft_consensus.cc:2454] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a4b49251da450988f5244bfb21462c in term 1.
I20260927 07:03:28.496630 22631 raft_consensus.cc:2454] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a4b49251da450988f5244bfb21462c in term 1.
I20260927 07:03:28.496781 22834 leader_election.cc:302] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3bef6edacb4ea69409638826470d11, 96a4b49251da450988f5244bfb21462c; no voters:
I20260927 07:03:28.496870 22967 raft_consensus.cc:2781] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:28.496995 22967 raft_consensus.cc:686] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 1 LEADER]: Becoming Leader. State: Replica: 96a4b49251da450988f5244bfb21462c, State: Running, Role: LEADER
I20260927 07:03:28.497084 22967 consensus_queue.cc:237] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } }
I20260927 07:03:28.497831 22486 catalog_manager.cc:5495] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96a4b49251da450988f5244bfb21462c (127.3.57.3). New cstate: current_term: 1 leader_uuid: "96a4b49251da450988f5244bfb21462c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96a4b49251da450988f5244bfb21462c" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 46211 } health_report { overall_health: HEALTHY } } }
I20260927 07:03:28.508981 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.506621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.3.57.3:46211)
0927 07:03:28.506681 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.506683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.506692 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:28.506774 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.506775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.506792 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.506864 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:28.506995 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.507439 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.508202 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.508406 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.508476 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.508669 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:28.508768 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:28.508864 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.508882 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.508899 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:03:28.510603 22765 raft_consensus.cc:1261] T 35c4921e3fc04a3686365dc7b4fa22ae P 0ffe1093d08f4375a244adc0f3eec3c6 [term 1 FOLLOWER]: Refusing update from remote peer 96a4b49251da450988f5244bfb21462c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:28.510613 22631 raft_consensus.cc:1261] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 1 FOLLOWER]: Refusing update from remote peer 96a4b49251da450988f5244bfb21462c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:28.510931 22967 consensus_queue.cc:1035] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ffe1093d08f4375a244adc0f3eec3c6" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:28.511075 22969 consensus_queue.cc:1035] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3bef6edacb4ea69409638826470d11" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:28.513422 22972 mvcc.cc:205] Tried to move back new op lower bound from 7333857724457185280 to 7333857724404781056. Current Snapshot: MvccSnapshot[applied={T|T < 7333857724457185280}]
I20260927 07:03:28.514025 22974 mvcc.cc:205] Tried to move back new op lower bound from 7333857724457185280 to 7333857724404781056. Current Snapshot: MvccSnapshot[applied={T|T < 7333857724457185280}]
I20260927 07:03:28.514087 22973 mvcc.cc:205] Tried to move back new op lower bound from 7333857724457185280 to 7333857724404781056. Current Snapshot: MvccSnapshot[applied={T|T < 7333857724457185280}]
I20260927 07:03:28.515638 22485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:28.515743 22485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:28.516441 22485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ac2fb4484b34e8db9e64ae085e8e944: Sending DeleteTablet for 3 replicas of tablet 35c4921e3fc04a3686365dc7b4fa22ae
I20260927 07:03:28.516768 22745 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:59322
I20260927 07:03:28.516788 22879 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:42526
I20260927 07:03:28.516773 22611 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c4921e3fc04a3686365dc7b4fa22ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:28 UTC) from {username='kudu', principal='kudu/127.3.57.62@KRBTEST.COM'} at 127.0.0.1:43890
I20260927 07:03:28.517017 22983 tablet_replica.cc:330] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11: stopping tablet replica
I20260927 07:03:28.517119 22983 raft_consensus.cc:2229] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:28.517217 22983 raft_consensus.cc:2258] T 35c4921e3fc04a3686365dc7b4fa22ae P 6f3bef6edacb4ea69409638826470d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:28.517247 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22548
W20260927 07:03:28.523188 22472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:44659 (error 108)
I20260927 07:03:28.523689 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.523578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:44659 (local address 127.0.0.1:43894)
0927 07:03:28.523615 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.523656 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:44659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 07:03:28.523772 22984 tablet_replica.cc:330] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: stopping tablet replica
I20260927 07:03:28.523761 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22682
I20260927 07:03:28.523866 22984 raft_consensus.cc:2229] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:03:28.523986 22984 raft_consensus.cc:2258] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:28.524561 22984 ts_tablet_manager.cc:1905] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:28.526441 22984 ts_tablet_manager.cc:1918] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:28.526515 22984 log.cc:1196] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/35c4921e3fc04a3686365dc7b4fa22ae
I20260927 07:03:28.526796 22984 ts_tablet_manager.cc:1939] T 35c4921e3fc04a3686365dc7b4fa22ae P 96a4b49251da450988f5244bfb21462c: Deleting consensus metadata
W20260927 07:03:28.527226 22472 catalog_manager.cc:4584] TS 6f3bef6edacb4ea69409638826470d11 (127.3.57.1:44659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35c4921e3fc04a3686365dc7b4fa22ae: Network error: Client connection negotiation failed: client connection to 127.3.57.1:44659: connect: Connection refused (error 111)
I20260927 07:03:28.527347 22472 catalog_manager.cc:4841] TS 96a4b49251da450988f5244bfb21462c (127.3.57.3:46211): tablet 35c4921e3fc04a3686365dc7b4fa22ae (table test-table [id=4db91709863d425f96fa50aea3c425a6]) successfully deleted
I20260927 07:03:28.530493 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22816
I20260927 07:03:28.530508 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.530391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:44381 (local address 127.0.0.1:59328)
0927 07:03:28.530449 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.530480 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 07:03:28.534672 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22454
2026-09-27T07:03:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1890 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T07:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:28Z Disabled control of system clock
I20260927 07:03:28.561254 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41291
--webserver_interface=127.3.57.62
--webserver_port=0
--builtin_ntp_servers=127.3.57.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:28.638324 22989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.638515 22989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.638536 22989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.639485 22989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:28.639516 22989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.639528 22989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.639539 22989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:28.639590 22989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:28.640748 22989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.640946 22989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.641156 22989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.645572 22994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.645608 22995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.645579 22997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.645808 22989 server_base.cc:1034] running on GCE node
I20260927 07:03:28.646049 22989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.646292 22989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.647445 22989 hybrid_clock.cc:648] HybridClock initialized: now 1790492608647434 us; error 28 us; skew 500 ppm
I20260927 07:03:28.648561 22989 webserver.cc:460] Webserver started at http://127.3.57.62:33399/ using document root <none> and password file <none>
I20260927 07:03:28.648780 22989 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.648831 22989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.649003 22989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.649967 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "982db4f08c21408ca79fee68d774a664"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "dba3011dc7eda605e021019d4c82a878"
server_key_iv: "668dfe5374cbe779640dac5c93bd449d"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.650300 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "982db4f08c21408ca79fee68d774a664"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "dba3011dc7eda605e021019d4c82a878"
server_key_iv: "668dfe5374cbe779640dac5c93bd449d"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.651469 22989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:28.652164 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.652319 22989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.652413 22989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "982db4f08c21408ca79fee68d774a664"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "dba3011dc7eda605e021019d4c82a878"
server_key_iv: "668dfe5374cbe779640dac5c93bd449d"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.652493 22989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.662057 22989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.662310 22989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.662452 22989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.666178 22989 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41291
I20260927 07:03:28.666208 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41291 every 8 connection(s)
I20260927 07:03:28.666497 22989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:28.667356 23056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:28.669431 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: Bootstrap starting.
I20260927 07:03:28.669927 23056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:28.670156 23056 log.cc:824] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:28.670697 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: No bootstrap required, opened a new log
I20260927 07:03:28.672274 23056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } }
I20260927 07:03:28.672396 23056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:28.672417 23056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982db4f08c21408ca79fee68d774a664, State: Initialized, Role: FOLLOWER
I20260927 07:03:28.672538 23056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } }
I20260927 07:03:28.672623 23056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:28.672645 23056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:28.672672 23056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:28.673117 23056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } }
I20260927 07:03:28.673192 23056 leader_election.cc:302] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 982db4f08c21408ca79fee68d774a664; no voters:
I20260927 07:03:28.673300 23056 leader_election.cc:288] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:28.673372 23059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:28.673528 23056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:28.673550 23059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [term 1 LEADER]: Becoming Leader. State: Replica: 982db4f08c21408ca79fee68d774a664, State: Running, Role: LEADER
I20260927 07:03:28.673666 23059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } }
I20260927 07:03:28.674031 23060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "982db4f08c21408ca79fee68d774a664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } } }
I20260927 07:03:28.674113 23060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:28.674122 23061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 982db4f08c21408ca79fee68d774a664. Latest consensus state: current_term: 1 leader_uuid: "982db4f08c21408ca79fee68d774a664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982db4f08c21408ca79fee68d774a664" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41291 } } }
I20260927 07:03:28.674219 23061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:28.674719 23069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:28.675230 23069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:28.675823 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 22989
I20260927 07:03:28.675937 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:28.676110 3300 external_mini_cluster.cc:1428] Setting key f1892b37edc78c2fca0b2bb766a88252
I20260927 07:03:28.676712 23069 catalog_manager.cc:1347] Generated new cluster ID: cacbdd614d6d4d5c83b200de971cd587
I20260927 07:03:28.676756 23069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:28.683871 23069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:28.684536 23069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:28.687667 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.676741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.3.57.62:41291)
0927 07:03:28.676917 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.676930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.683369 (+ 6439us) server_negotiation.cc:407] Connection header received
0927 07:03:28.683508 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.683517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.683640 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.683808 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.684050 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.685028 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.685769 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.686172 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.686294 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.686942 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.686946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.686946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:28.687109 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.687137 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:28.687146 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.687239 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.687312 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.687409 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260927 07:03:28.690589 23069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: Generated new TSK 0
I20260927 07:03:28.690725 23069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:28.694336 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.1:0
--local_ip_for_outbound_sockets=127.3.57.1
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--builtin_ntp_servers=127.3.57.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:03:28.781400 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.781574 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.781592 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.782678 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.782716 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.782739 23080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.1
I20260927 07:03:28.784119 23080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.57.1
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.1
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.784328 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.784639 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.789486 23086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.789454 23088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.789614 23085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.789614 23080 server_base.cc:1034] running on GCE node
I20260927 07:03:28.789834 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.790079 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.791244 23080 hybrid_clock.cc:648] HybridClock initialized: now 1790492608791222 us; error 40 us; skew 500 ppm
I20260927 07:03:28.792676 23080 webserver.cc:460] Webserver started at http://127.3.57.1:42001/ using document root <none> and password file <none>
I20260927 07:03:28.792932 23080 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.793000 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.793134 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.794138 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data/instance:
uuid: "dc0b4f3bd223420693563ec115d1144d"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4518a89cebc3b7b4488b6b68a3fc80f5"
server_key_iv: "0851e19c441ee4a896f9bd17f3238f35"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.794479 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance:
uuid: "dc0b4f3bd223420693563ec115d1144d"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4518a89cebc3b7b4488b6b68a3fc80f5"
server_key_iv: "0851e19c441ee4a896f9bd17f3238f35"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.795778 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:28.796484 23094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.796648 23080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.796746 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal
uuid: "dc0b4f3bd223420693563ec115d1144d"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4518a89cebc3b7b4488b6b68a3fc80f5"
server_key_iv: "0851e19c441ee4a896f9bd17f3238f35"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.796830 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.811167 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.811431 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.811630 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.811833 23080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:28.812124 23080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:28.812187 23080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.812225 23080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:28.812252 23080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.817500 23080 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.1:46405
I20260927 07:03:28.817575 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.1:46405 every 8 connection(s)
I20260927 07:03:28.817874 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/data/info.pb
I20260927 07:03:28.819199 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23080
I20260927 07:03:28.819299 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-0/wal/instance
I20260927 07:03:28.819489 3300 external_mini_cluster.cc:1428] Setting key 6f3282b6c1e99d9e62a1414289d6aadf
I20260927 07:03:28.820506 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.2:0
--local_ip_for_outbound_sockets=127.3.57.2
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--builtin_ntp_servers=127.3.57.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:03:28.823997 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.818632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.1:50773 (local address 127.3.57.62:41291)
0927 07:03:28.818702 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.818704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.818969 (+ 265us) server_negotiation.cc:407] Connection header received
0927 07:03:28.819709 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.819710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.819726 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.819803 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.820680 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.821255 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.822408 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.822738 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.822818 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.823666 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.823667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.823667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:28.823672 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.823689 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:28.823694 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.823763 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.823854 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.823889 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 07:03:28.824185 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.818632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41291 (local address 127.3.57.1:50773)
0927 07:03:28.818806 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.818851 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.819514 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.819902 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.819906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.819916 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:28.820556 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.820564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.821351 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.821353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.822315 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.822317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.822829 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.822830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.822939 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.823477 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:28.823477 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:28.823503 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:28.823523 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:28.823770 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:28.823774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:28.823776 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.823830 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.823932 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260927 07:03:28.824663 23208 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41291
I20260927 07:03:28.824765 23208 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:28.824965 23208 heartbeater.cc:511] Master 127.3.57.62:41291 requested a full tablet report, sending...
I20260927 07:03:28.825436 23020 ts_manager.cc:194] Registered new tserver with Master: dc0b4f3bd223420693563ec115d1144d (127.3.57.1:46405)
I20260927 07:03:28.826068 23020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.1:50773
W20260927 07:03:28.900043 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:28.900261 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:28.900305 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:28.901444 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:28.901481 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:28.901506 23211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.2
I20260927 07:03:28.902779 23211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.57.2
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.2
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:28.902941 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:28.903162 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:28.907811 23217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.907868 23216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:28.907826 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:28.907966 23211 server_base.cc:1034] running on GCE node
I20260927 07:03:28.908288 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:28.908510 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:28.909691 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790492608909685 us; error 35 us; skew 500 ppm
I20260927 07:03:28.911108 23211 webserver.cc:460] Webserver started at http://127.3.57.2:41179/ using document root <none> and password file <none>
I20260927 07:03:28.911368 23211 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:28.911424 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:28.911537 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:28.912714 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data/instance:
uuid: "c321a015c1904871866115d0e6587dd4"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4d299ba25db48dee7302d50908f63c72"
server_key_iv: "281bab0dc13c54b76f1ff49abfb82d86"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.913028 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance:
uuid: "c321a015c1904871866115d0e6587dd4"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4d299ba25db48dee7302d50908f63c72"
server_key_iv: "281bab0dc13c54b76f1ff49abfb82d86"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.914225 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:03:28.914920 23225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.915138 23211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:03:28.915231 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal
uuid: "c321a015c1904871866115d0e6587dd4"
format_stamp: "Formatted at 2026-09-27 07:03:28 on dist-test-slave-mk3h"
server_key: "4d299ba25db48dee7302d50908f63c72"
server_key_iv: "281bab0dc13c54b76f1ff49abfb82d86"
server_key_version: "encryptionkey@0"
I20260927 07:03:28.915318 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:28.925401 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:28.925649 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:28.925789 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:28.925997 23211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:28.926298 23211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:28.926353 23211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.926391 23211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:28.926419 23211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:28.931996 23211 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.2:38299
I20260927 07:03:28.932119 23338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.2:38299 every 8 connection(s)
I20260927 07:03:28.932377 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/data/info.pb
I20260927 07:03:28.935388 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23211
I20260927 07:03:28.935513 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-1/wal/instance
I20260927 07:03:28.935720 3300 external_mini_cluster.cc:1428] Setting key 6703b188779ea7c45928ff2322dc1658
I20260927 07:03:28.936852 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.57.3:0
--local_ip_for_outbound_sockets=127.3.57.3
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--builtin_ntp_servers=127.3.57.20:39643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:03:28.937799 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.933094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.2:60373 (local address 127.3.57.62:41291)
0927 07:03:28.933165 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:28.933167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:28.933337 (+ 170us) server_negotiation.cc:407] Connection header received
0927 07:03:28.933787 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:28.933788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:28.933801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:28.933852 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:28.934475 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.934960 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.936161 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.936454 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.936515 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.937523 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:28.937525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:28.937525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:28.937531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:28.937563 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:28.937568 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:28.937596 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:28.937717 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 07:03:28.937743 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 07:03:28.938059 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:28.933047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41291 (local address 127.3.57.2:60373)
0927 07:03:28.933225 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:28.933256 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:28.933706 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:28.933902 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:28.933905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:28.933913 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:28.934408 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.934412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.935062 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.935062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.936054 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:28.936057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:28.936541 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:28.936542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:28.936695 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:28.937277 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:28.937277 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:28.937314 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:28.937350 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:28.937639 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:28.937645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:28.937648 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:28.937709 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 07:03:28.937813 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260927 07:03:28.938479 23339 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41291
I20260927 07:03:28.938563 23339 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:28.938745 23339 heartbeater.cc:511] Master 127.3.57.62:41291 requested a full tablet report, sending...
I20260927 07:03:28.939105 23020 ts_manager.cc:194] Registered new tserver with Master: c321a015c1904871866115d0e6587dd4 (127.3.57.2:38299)
I20260927 07:03:28.939499 23020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.2:60373
W20260927 07:03:29.014757 23342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:29.014945 23342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:29.014964 23342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:29.016069 23342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:29.016119 23342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:29.016144 23342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.57.3
I20260927 07:03:29.017401 23342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.57.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.57.3
--webserver_port=0
--tserver_master_addrs=127.3.57.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.57.3
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:29.017634 23342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:29.017889 23342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:29.022555 23348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:29.022720 23342 server_base.cc:1034] running on GCE node
W20260927 07:03:29.022769 23347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:29.022826 23350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:29.023078 23342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:29.023308 23342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:29.024448 23342 hybrid_clock.cc:648] HybridClock initialized: now 1790492609024434 us; error 32 us; skew 500 ppm
I20260927 07:03:29.025596 23342 webserver.cc:460] Webserver started at http://127.3.57.3:39617/ using document root <none> and password file <none>
I20260927 07:03:29.025842 23342 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:29.025885 23342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:29.026026 23342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:29.026978 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data/instance:
uuid: "1e351e486a1b4fcca85b08a838271ce9"
format_stamp: "Formatted at 2026-09-27 07:03:29 on dist-test-slave-mk3h"
server_key: "a54d3c8bd85fdf7106546e0672a4f031"
server_key_iv: "15778f57bd66b4edcd2df9834e945ef4"
server_key_version: "encryptionkey@0"
I20260927 07:03:29.027310 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance:
uuid: "1e351e486a1b4fcca85b08a838271ce9"
format_stamp: "Formatted at 2026-09-27 07:03:29 on dist-test-slave-mk3h"
server_key: "a54d3c8bd85fdf7106546e0672a4f031"
server_key_iv: "15778f57bd66b4edcd2df9834e945ef4"
server_key_version: "encryptionkey@0"
I20260927 07:03:29.028642 23342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:29.029357 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:29.029542 23342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:29.029629 23342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal
uuid: "1e351e486a1b4fcca85b08a838271ce9"
format_stamp: "Formatted at 2026-09-27 07:03:29 on dist-test-slave-mk3h"
server_key: "a54d3c8bd85fdf7106546e0672a4f031"
server_key_iv: "15778f57bd66b4edcd2df9834e945ef4"
server_key_version: "encryptionkey@0"
I20260927 07:03:29.029697 23342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:29.049814 23342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:29.050077 23342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:29.050210 23342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:29.050431 23342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:03:29.050733 23342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:03:29.050791 23342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:29.050849 23342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:03:29.050885 23342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:29.056685 23342 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.3:33629
I20260927 07:03:29.056730 23469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.3:33629 every 8 connection(s)
I20260927 07:03:29.057015 23342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/data/info.pb
I20260927 07:03:29.061733 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.057665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:59347 (local address 127.3.57.62:41291)
0927 07:03:29.057731 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.057733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.057989 (+ 256us) server_negotiation.cc:407] Connection header received
0927 07:03:29.058421 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.058422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.058436 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.058488 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:29.059142 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.059694 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.060567 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.060742 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.060795 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.061536 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:29.061536 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:29.061537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:29.061540 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:29.061550 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:29.061553 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:29.061590 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.061656 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.061677 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:03:29.061939 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.057687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:41291 (local address 127.3.57.3:59347)
0927 07:03:29.057873 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.057903 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.058340 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.058558 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.058576 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.058584 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:29.059064 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.059067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.059766 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.059766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.060493 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.060495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.060820 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.060821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.060921 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.061424 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:03:29.061424 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:29.061450 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:03:29.061469 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:29.061598 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:29.061599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:29.061602 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.061646 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.061713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260927 07:03:29.062288 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23342
I20260927 07:03:29.062407 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal/instance
I20260927 07:03:29.062505 23470 heartbeater.cc:347] Connected to a master server at 127.3.57.62:41291
I20260927 07:03:29.062587 23470 heartbeater.cc:464] Registering TS with master...
I20260927 07:03:29.062605 3300 external_mini_cluster.cc:1428] Setting key 8f6716a1f275f55b2c7e442c588eda1b
I20260927 07:03:29.062768 23470 heartbeater.cc:511] Master 127.3.57.62:41291 requested a full tablet report, sending...
I20260927 07:03:29.063125 23020 ts_manager.cc:194] Registered new tserver with Master: 1e351e486a1b4fcca85b08a838271ce9 (127.3.57.3:33629)
I20260927 07:03:29.063433 3300 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:03:29.063622 23020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.57.3:59347
I20260927 07:03:29.067078 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.064774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.3.57.62:41291)
0927 07:03:29.064833 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.064835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.064907 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:03:29.064967 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.064968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.064974 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.065042 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:29.065178 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.065588 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.066231 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.066395 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.066439 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.066930 (+ 491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:29.066931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:29.066931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:29.066934 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:29.066944 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:29.066947 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:29.066986 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.067018 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.067033 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:03:29.068391 23020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:03:29.069907 23020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:03:29.076180 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.072133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:46405 (local address 127.0.0.1:47112)
0927 07:03:29.072329 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.072337 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.072372 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.072662 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.072665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.072667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:29.072726 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.072729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.073866 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.073867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.075081 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.075083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.075905 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.075906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.075977 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.075989 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.076062 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.076110 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":22,"thread_start_us":56,"threads_started":1}
I20260927 07:03:29.076195 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.072298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.3.57.1:46405)
0927 07:03:29.072460 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.072465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.072475 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:29.072517 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.072519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.072535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.072618 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:29.072821 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.073771 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.075165 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.075817 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.075905 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.075989 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.076075 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.076111 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260927 07:03:29.076416 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.071934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.3:33629 (local address 127.0.0.1:33464)
0927 07:03:29.072261 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.072270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.072313 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.073004 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.073005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.073008 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:29.073043 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.073044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.074268 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.074269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.075319 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.075322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.076173 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.076175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.076250 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.076254 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.076290 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.076352 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":67,"threads_started":1}
I20260927 07:03:29.076663 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.071982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:38299 (local address 127.0.0.1:38234)
0927 07:03:29.072123 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.072165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.072224 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.072706 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.072707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.072709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:29.072753 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.072755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.074123 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.074124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.075260 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.075261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.076175 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.076175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.076515 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.076520 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.076563 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.076597 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":38,"thread_start_us":79,"threads_started":1}
I20260927 07:03:29.076762 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.072100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.3.57.3:33629)
0927 07:03:29.072811 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.072815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.072824 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:29.072866 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.072868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.072914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.072991 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:29.073120 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.074188 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.075399 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.076069 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.076539 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.076624 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.076647 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.076684 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":82,"threads_started":1}
I20260927 07:03:29.076988 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.072094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.3.57.2:38299)
0927 07:03:29.072488 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.072492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.072501 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:29.072537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.072539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.072553 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.072703 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:29.072838 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.074043 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.075340 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.076068 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.076735 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.076841 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.076867 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.076906 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":101,"threads_started":1}
I20260927 07:03:29.076864 23142 tablet_service.cc:1467] Processing CreateTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e (DEFAULT_TABLE table=test-table [id=ac894aa858f54293985258553f523bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:29.077217 23142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399ac9e252d4c8fbcb75aac1624533e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:29.077258 23404 tablet_service.cc:1467] Processing CreateTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e (DEFAULT_TABLE table=test-table [id=ac894aa858f54293985258553f523bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:29.077482 23404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399ac9e252d4c8fbcb75aac1624533e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:29.079483 23484 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: Bootstrap starting.
I20260927 07:03:29.080008 23485 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Bootstrap starting.
I20260927 07:03:29.080050 23273 tablet_service.cc:1467] Processing CreateTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e (DEFAULT_TABLE table=test-table [id=ac894aa858f54293985258553f523bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:03:29.080363 23484 tablet_bootstrap.cc:654] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:29.080375 23273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0399ac9e252d4c8fbcb75aac1624533e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:29.080595 23485 tablet_bootstrap.cc:654] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:29.080672 23484 log.cc:824] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:29.080912 23485 log.cc:824] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:29.081432 23484 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: No bootstrap required, opened a new log
I20260927 07:03:29.081503 23484 ts_tablet_manager.cc:1397] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:29.081662 23485 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: No bootstrap required, opened a new log
I20260927 07:03:29.081723 23485 ts_tablet_manager.cc:1397] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:29.083370 23488 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Bootstrap starting.
I20260927 07:03:29.083935 23488 tablet_bootstrap.cc:654] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:29.084209 23488 log.cc:824] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:29.084218 23484 raft_consensus.cc:348] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.084218 23485 raft_consensus.cc:348] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.084379 23484 raft_consensus.cc:374] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:29.084379 23485 raft_consensus.cc:374] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:29.084425 23484 raft_consensus.cc:729] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0b4f3bd223420693563ec115d1144d, State: Initialized, Role: FOLLOWER
I20260927 07:03:29.084425 23485 raft_consensus.cc:729] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e351e486a1b4fcca85b08a838271ce9, State: Initialized, Role: FOLLOWER
I20260927 07:03:29.084535 23484 consensus_queue.cc:260] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.084535 23485 consensus_queue.cc:260] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.084784 23485 ts_tablet_manager.cc:1428] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:03:29.084784 23484 ts_tablet_manager.cc:1428] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:03:29.084831 23208 heartbeater.cc:503] Master 127.3.57.62:41291 was elected leader, sending a full tablet report...
I20260927 07:03:29.084889 23470 heartbeater.cc:503] Master 127.3.57.62:41291 was elected leader, sending a full tablet report...
I20260927 07:03:29.085347 23488 tablet_bootstrap.cc:492] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: No bootstrap required, opened a new log
I20260927 07:03:29.085409 23488 ts_tablet_manager.cc:1397] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:03:29.087201 23488 raft_consensus.cc:348] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.087316 23488 raft_consensus.cc:374] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:29.087388 23488 raft_consensus.cc:729] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c321a015c1904871866115d0e6587dd4, State: Initialized, Role: FOLLOWER
I20260927 07:03:29.087504 23488 consensus_queue.cc:260] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.087755 23488 ts_tablet_manager.cc:1428] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:03:29.087826 23339 heartbeater.cc:503] Master 127.3.57.62:41291 was elected leader, sending a full tablet report...
W20260927 07:03:29.182910 23340 tablet.cc:2369] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:03:29.245316 23490 raft_consensus.cc:482] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:03:29.245428 23490 raft_consensus.cc:504] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.245853 23490 leader_election.cc:288] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0b4f3bd223420693563ec115d1144d (127.3.57.1:46405), c321a015c1904871866115d0e6587dd4 (127.3.57.2:38299)
I20260927 07:03:29.248677 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.246078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:38971 (local address 127.3.57.1:46405)
0927 07:03:29.246126 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.246128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.246141 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:03:29.246223 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.246224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.246236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.246314 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:29.246461 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.246951 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.247965 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.248339 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.248422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.248431 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.248570 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.248606 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 07:03:29.248710 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.246010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:46405 (local address 127.3.57.3:38971)
0927 07:03:29.246101 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.246113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.246160 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.246320 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.246321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.246322 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:29.246393 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.246395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.247033 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.247033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.247870 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.247872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.248421 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.248422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.248489 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.248492 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.248562 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.248602 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":30}
I20260927 07:03:29.248715 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.246121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.57.3:55675 (local address 127.3.57.2:38299)
0927 07:03:29.246160 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.246162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.246289 (+ 127us) server_negotiation.cc:407] Connection header received
0927 07:03:29.246363 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.246364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.246378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.246441 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:03:29.246587 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.247070 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.248012 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.248367 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.248448 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.248458 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.248606 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.248643 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:29.248929 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.246051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:38299 (local address 127.3.57.3:55675)
0927 07:03:29.246210 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.246218 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:29.246292 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:29.246443 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:29.246444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:29.246445 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:03:29.246519 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.246520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.247154 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.247155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.247919 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:29.247921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.248462 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.248463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:29.248532 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.248535 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 07:03:29.248599 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 07:03:29.248638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260927 07:03:29.249099 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399ac9e252d4c8fbcb75aac1624533e" candidate_uuid: "1e351e486a1b4fcca85b08a838271ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0b4f3bd223420693563ec115d1144d" is_pre_election: true
I20260927 07:03:29.249255 23162 raft_consensus.cc:2454] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e351e486a1b4fcca85b08a838271ce9 in term 0.
I20260927 07:03:29.249316 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399ac9e252d4c8fbcb75aac1624533e" candidate_uuid: "1e351e486a1b4fcca85b08a838271ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c321a015c1904871866115d0e6587dd4" is_pre_election: true
I20260927 07:03:29.249519 23293 raft_consensus.cc:2454] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e351e486a1b4fcca85b08a838271ce9 in term 0.
I20260927 07:03:29.249549 23357 leader_election.cc:302] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e351e486a1b4fcca85b08a838271ce9, dc0b4f3bd223420693563ec115d1144d; no voters:
I20260927 07:03:29.249756 23490 raft_consensus.cc:2781] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:03:29.249799 23490 raft_consensus.cc:482] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:03:29.249816 23490 raft_consensus.cc:3037] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:29.250464 23490 raft_consensus.cc:504] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.250590 23490 leader_election.cc:288] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [CANDIDATE]: Term 1 election: Requested vote from peers dc0b4f3bd223420693563ec115d1144d (127.3.57.1:46405), c321a015c1904871866115d0e6587dd4 (127.3.57.2:38299)
I20260927 07:03:29.250730 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399ac9e252d4c8fbcb75aac1624533e" candidate_uuid: "1e351e486a1b4fcca85b08a838271ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0b4f3bd223420693563ec115d1144d"
I20260927 07:03:29.250725 23293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0399ac9e252d4c8fbcb75aac1624533e" candidate_uuid: "1e351e486a1b4fcca85b08a838271ce9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c321a015c1904871866115d0e6587dd4"
I20260927 07:03:29.250805 23293 raft_consensus.cc:3037] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:29.250813 23162 raft_consensus.cc:3037] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:29.251662 23293 raft_consensus.cc:2454] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e351e486a1b4fcca85b08a838271ce9 in term 1.
I20260927 07:03:29.251662 23162 raft_consensus.cc:2454] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e351e486a1b4fcca85b08a838271ce9 in term 1.
I20260927 07:03:29.251818 23357 leader_election.cc:302] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e351e486a1b4fcca85b08a838271ce9, dc0b4f3bd223420693563ec115d1144d; no voters:
I20260927 07:03:29.251930 23490 raft_consensus.cc:2781] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:29.252064 23490 raft_consensus.cc:686] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 1e351e486a1b4fcca85b08a838271ce9, State: Running, Role: LEADER
I20260927 07:03:29.252162 23490 consensus_queue.cc:237] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } }
I20260927 07:03:29.252965 23020 catalog_manager.cc:5495] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e351e486a1b4fcca85b08a838271ce9 (127.3.57.3). New cstate: current_term: 1 leader_uuid: "1e351e486a1b4fcca85b08a838271ce9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e351e486a1b4fcca85b08a838271ce9" member_type: VOTER last_known_addr { host: "127.3.57.3" port: 33629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 } health_report { overall_health: UNKNOWN } } }
I20260927 07:03:29.300410 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.297880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.3.57.3:33629)
0927 07:03:29.297956 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:29.297958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:29.297968 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:29.298033 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:29.298036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:29.298057 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:29.298136 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:29.298306 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.298786 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.299581 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:29.299789 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:29.299884 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:29.300062 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:29.300165 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:29.300260 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:29.300279 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 07:03:29.300296 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:03:29.302311 23162 raft_consensus.cc:1261] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 1 FOLLOWER]: Refusing update from remote peer 1e351e486a1b4fcca85b08a838271ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:29.302311 23293 raft_consensus.cc:1261] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 1 FOLLOWER]: Refusing update from remote peer 1e351e486a1b4fcca85b08a838271ce9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:29.302706 23494 consensus_queue.cc:1035] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c321a015c1904871866115d0e6587dd4" member_type: VOTER last_known_addr { host: "127.3.57.2" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:29.302857 23490 consensus_queue.cc:1035] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0b4f3bd223420693563ec115d1144d" member_type: VOTER last_known_addr { host: "127.3.57.1" port: 46405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:29.304915 23497 mvcc.cc:205] Tried to move back new op lower bound from 7333857727699226624 to 7333857727497621504. Current Snapshot: MvccSnapshot[applied={T|T < 7333857727699226624}]
I20260927 07:03:29.305297 23499 mvcc.cc:205] Tried to move back new op lower bound from 7333857727699226624 to 7333857727497621504. Current Snapshot: MvccSnapshot[applied={T|T < 7333857727699226624}]
I20260927 07:03:29.307714 23019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:29.307827 23019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:03:29.308537 23019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 982db4f08c21408ca79fee68d774a664: Sending DeleteTablet for 3 replicas of tablet 0399ac9e252d4c8fbcb75aac1624533e
I20260927 07:03:29.308866 23273 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:29 UTC) from {username='slave'} at 127.0.0.1:38234
I20260927 07:03:29.308877 23404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:29 UTC) from {username='slave'} at 127.0.0.1:33464
I20260927 07:03:29.308929 23142 tablet_service.cc:1514] Processing DeleteTablet for tablet 0399ac9e252d4c8fbcb75aac1624533e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:03:29 UTC) from {username='slave'} at 127.0.0.1:47112
I20260927 07:03:29.309116 23509 tablet_replica.cc:330] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d: stopping tablet replica
I20260927 07:03:29.309230 23509 raft_consensus.cc:2229] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:29.309259 23508 tablet_replica.cc:330] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4: stopping tablet replica
I20260927 07:03:29.309329 23509 raft_consensus.cc:2258] T 0399ac9e252d4c8fbcb75aac1624533e P dc0b4f3bd223420693563ec115d1144d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:29.309346 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23080
W20260927 07:03:29.313419 23004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.57.1:46405 (error 108)
I20260927 07:03:29.313925 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.313782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.1:46405 (local address 127.0.0.1:47114)
0927 07:03:29.313847 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.313887 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.1:46405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 07:03:29.314039 23004 catalog_manager.cc:4584] TS dc0b4f3bd223420693563ec115d1144d (127.3.57.1:46405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0399ac9e252d4c8fbcb75aac1624533e: Network error: Client connection negotiation failed: client connection to 127.3.57.1:46405: connect: Connection refused (error 111)
I20260927 07:03:29.314776 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23211
I20260927 07:03:29.314788 23508 raft_consensus.cc:2229] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:03:29.314898 23508 raft_consensus.cc:2258] T 0399ac9e252d4c8fbcb75aac1624533e P c321a015c1904871866115d0e6587dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:29.315047 23507 tablet_replica.cc:330] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: stopping tablet replica
I20260927 07:03:29.315124 23507 raft_consensus.cc:2229] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:03:29.315219 23507 raft_consensus.cc:2258] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:03:29.316176 23507 ts_tablet_manager.cc:1905] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:03:29.317648 23507 ts_tablet_manager.cc:1918] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:03:29.317739 23507 log.cc:1196] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Deleting WAL directory at /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492341973781-3300-0/minicluster-data/ts-2/wal/wals/0399ac9e252d4c8fbcb75aac1624533e
I20260927 07:03:29.317989 23507 ts_tablet_manager.cc:1939] T 0399ac9e252d4c8fbcb75aac1624533e P 1e351e486a1b4fcca85b08a838271ce9: Deleting consensus metadata
I20260927 07:03:29.318356 23004 catalog_manager.cc:4841] TS 1e351e486a1b4fcca85b08a838271ce9 (127.3.57.3:33629): tablet 0399ac9e252d4c8fbcb75aac1624533e (table test-table [id=ac894aa858f54293985258553f523bbe]) successfully deleted
I20260927 07:03:29.321527 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:29.321426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.2:38299 (local address 127.0.0.1:38246)
0927 07:03:29.321473 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:29.321497 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.2:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 07:03:29.321570 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23342
I20260927 07:03:29.326153 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 22989
2026-09-27T07:03:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (790 ms)
[----------] 6 tests from EncryptionPolicyTest (8020 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:29 dist-test-slave-mk3h krb5kdc[23513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:29 dist-test-slave-mk3h krb5kdc[23513](info): set up 2 sockets
Sep 27 07:03:29 dist-test-slave-mk3h krb5kdc[23513](info): commencing operation
krb5kdc: starting...
W20260927 07:03:30.355396 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:30.466261 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:30.544305 23526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:30.544529 23526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:30.544566 23526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:30.545701 23526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:30.545782 23526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:30.545805 23526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:30.545832 23526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:30.545872 23526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:30.547266 23526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:30.547524 23526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:30.547789 23526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:30.552418 23534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:30.552572 23531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:30.552644 23532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:30.552675 23526 server_base.cc:1034] running on GCE node
I20260927 07:03:30.552877 23526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:30.553100 23526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:30.554232 23526 hybrid_clock.cc:648] HybridClock initialized: now 1790492610554223 us; error 27 us; skew 500 ppm
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:30.556587 23526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:30.556983 23526 webserver.cc:460] Webserver started at http://127.0.0.1:37709/ using document root <none> and password file <none>
I20260927 07:03:30.557194 23526 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:30.557237 23526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:30.557384 23526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:30.558387 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "6b3a7003bec6497082ace58d4ca2fc17"
format_stamp: "Formatted at 2026-09-27 07:03:30 on dist-test-slave-mk3h"
server_key: "531920c206c6cde11575debf750ea12f"
server_key_iv: "829810cdc8fc2d180fe66c453e25ed6a"
server_key_version: "encryptionkey@0"
I20260927 07:03:30.558735 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "6b3a7003bec6497082ace58d4ca2fc17"
format_stamp: "Formatted at 2026-09-27 07:03:30 on dist-test-slave-mk3h"
server_key: "531920c206c6cde11575debf750ea12f"
server_key_iv: "829810cdc8fc2d180fe66c453e25ed6a"
server_key_version: "encryptionkey@0"
I20260927 07:03:30.560117 23526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 07:03:30.560779 23541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:30.560993 23526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:30.561077 23526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "6b3a7003bec6497082ace58d4ca2fc17"
format_stamp: "Formatted at 2026-09-27 07:03:30 on dist-test-slave-mk3h"
server_key: "531920c206c6cde11575debf750ea12f"
server_key_iv: "829810cdc8fc2d180fe66c453e25ed6a"
server_key_version: "encryptionkey@0"
I20260927 07:03:30.561127 23526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:30.575114 23526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:30.576928 23526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:30.577073 23526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:30.580827 23526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42621
I20260927 07:03:30.580878 23593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42621 every 8 connection(s)
I20260927 07:03:30.581179 23526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:30.582090 23594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:30.584158 23594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17: Bootstrap starting.
I20260927 07:03:30.584650 23594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:30.584885 23594 log.cc:824] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:30.585552 23594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17: No bootstrap required, opened a new log
I20260927 07:03:30.587055 23594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } }
I20260927 07:03:30.587178 23594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:30.587200 23594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3a7003bec6497082ace58d4ca2fc17, State: Initialized, Role: FOLLOWER
I20260927 07:03:30.587307 23594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } }
I20260927 07:03:30.587381 23594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:30.587414 23594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:30.587482 23594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:30.588032 23594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } }
I20260927 07:03:30.588143 23594 leader_election.cc:302] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b3a7003bec6497082ace58d4ca2fc17; no voters:
I20260927 07:03:30.588308 23594 leader_election.cc:288] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:30.588356 23597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:30.588539 23594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:30.588562 23597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [term 1 LEADER]: Becoming Leader. State: Replica: 6b3a7003bec6497082ace58d4ca2fc17, State: Running, Role: LEADER
I20260927 07:03:30.588637 23597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } }
I20260927 07:03:30.589156 23599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b3a7003bec6497082ace58d4ca2fc17. Latest consensus state: current_term: 1 leader_uuid: "6b3a7003bec6497082ace58d4ca2fc17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } } }
I20260927 07:03:30.589257 23599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:30.589259 23598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b3a7003bec6497082ace58d4ca2fc17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b3a7003bec6497082ace58d4ca2fc17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42621 } } }
I20260927 07:03:30.589427 23598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:30.589932 23605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:30.590435 23605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:30.590510 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23526
I20260927 07:03:30.590608 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:30.590819 3300 external_mini_cluster.cc:1428] Setting key 79330ae82cece7cb3f5ff4955f248b05
I20260927 07:03:30.591991 23605 catalog_manager.cc:1347] Generated new cluster ID: 9d1870eac8a446b680e6f58625f9099c
I20260927 07:03:30.592041 23605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:03:30.606670 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:30.591254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50325 (local address 127.0.0.1:42621)
0927 07:03:30.591394 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:30.591403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:30.595798 (+ 4395us) server_negotiation.cc:407] Connection header received
0927 07:03:30.595940 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:30.595947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:30.595998 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:30.596132 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:30.596893 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:30.598513 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:30.599647 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:30.600219 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:30.600320 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:30.603050 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:30.603054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:30.603055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:30.603068 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:30.605030 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:30.605442 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:30.605445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:30.605445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:30.605552 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:30.605776 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:30.605777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:30.605777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:30.606095 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:30.606250 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:30.606342 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 07:03:30.606443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260927 07:03:30.607738 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:30.611231 23605 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:30.611815 23605 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:30.614704 23605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b3a7003bec6497082ace58d4ca2fc17: Generated new TSK 0
I20260927 07:03:30.614933 23605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:30.616155 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:30.609101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38933 (local address 127.0.0.1:42621)
0927 07:03:30.609143 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:30.609145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:30.609263 (+ 118us) server_negotiation.cc:407] Connection header received
0927 07:03:30.609308 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:30.609308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:30.609318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:30.609374 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:30.609976 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:30.610685 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:30.611770 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:30.612082 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:30.612148 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:30.613454 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:30.613455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:30.613455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:30.613461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:30.615124 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:30.615490 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:30.615490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:30.615491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:30.615534 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:30.615771 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:30.615772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:30.615772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:30.615863 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:30.615943 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:30.616068 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 07:03:30.616107 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 07:03:30.618084 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23526
W20260927 07:03:30.622339 23511 connection.cc:535] client connection to 127.0.0.1:42621 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42621): Connection reset by peer (error 104)
2026-09-27T07:03:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23625](info): set up 2 sockets
Sep 27 07:03:30 dist-test-slave-mk3h krb5kdc[23625](info): commencing operation
krb5kdc: starting...
W20260927 07:03:31.643375 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:31 dist-test-slave-mk3h krb5kdc[23625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:31.756562 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:31.834193 23638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:31.834426 23638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:31.834461 23638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:31.835683 23638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:31.835742 23638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:31.835767 23638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:31.835798 23638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:31.835825 23638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:31.837286 23638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:31.837563 23638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:31.837787 23638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:31.842648 23646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:31.842701 23638 server_base.cc:1034] running on GCE node
W20260927 07:03:31.842741 23643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:31.842811 23644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:31.843155 23638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:31.843441 23638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:31.844611 23638 hybrid_clock.cc:648] HybridClock initialized: now 1790492611844598 us; error 40 us; skew 500 ppm
Sep 27 07:03:31 dist-test-slave-mk3h krb5kdc[23625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:31.846824 23638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:31.847288 23638 webserver.cc:460] Webserver started at http://127.0.0.1:38853/ using document root <none> and password file <none>
I20260927 07:03:31.847509 23638 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:31.847602 23638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:31.847769 23638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:31.848728 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "2b93313a498d48eab540c057f32d828e"
format_stamp: "Formatted at 2026-09-27 07:03:31 on dist-test-slave-mk3h"
server_key: "53090ab8cc74da283934acedb5689459"
server_key_iv: "69ed5e11b6a806268fdb12034f11ff28"
server_key_version: "encryptionkey@0"
I20260927 07:03:31.849072 23638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "2b93313a498d48eab540c057f32d828e"
format_stamp: "Formatted at 2026-09-27 07:03:31 on dist-test-slave-mk3h"
server_key: "53090ab8cc74da283934acedb5689459"
server_key_iv: "69ed5e11b6a806268fdb12034f11ff28"
server_key_version: "encryptionkey@0"
I20260927 07:03:31.850360 23638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:31.851097 23653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:31.851269 23638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:31.851365 23638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "2b93313a498d48eab540c057f32d828e"
format_stamp: "Formatted at 2026-09-27 07:03:31 on dist-test-slave-mk3h"
server_key: "53090ab8cc74da283934acedb5689459"
server_key_iv: "69ed5e11b6a806268fdb12034f11ff28"
server_key_version: "encryptionkey@0"
I20260927 07:03:31.851446 23638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:31.867821 23638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:31.869653 23638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:31.869812 23638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:31.873847 23638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41757
I20260927 07:03:31.873879 23705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41757 every 8 connection(s)
I20260927 07:03:31.874210 23638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:31.875092 23706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:31.877323 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e: Bootstrap starting.
I20260927 07:03:31.877875 23706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:31.878147 23706 log.cc:824] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:31.878814 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e: No bootstrap required, opened a new log
I20260927 07:03:31.880424 23706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260927 07:03:31.880548 23706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:31.880529 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23638
I20260927 07:03:31.880569 23706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b93313a498d48eab540c057f32d828e, State: Initialized, Role: FOLLOWER
I20260927 07:03:31.880652 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:31.880723 23706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260927 07:03:31.880798 23706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:31.880856 23706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:31.880916 23706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:31.880939 3300 external_mini_cluster.cc:1428] Setting key 79232092e65ef002131e86c79f42be73
I20260927 07:03:31.882032 23706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260927 07:03:31.882134 23706 leader_election.cc:302] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b93313a498d48eab540c057f32d828e; no voters:
I20260927 07:03:31.882274 23706 leader_election.cc:288] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:31.882309 23711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:31.882467 23711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [term 1 LEADER]: Becoming Leader. State: Replica: 2b93313a498d48eab540c057f32d828e, State: Running, Role: LEADER
I20260927 07:03:31.882516 23706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:31.882581 23711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } }
I20260927 07:03:31.882966 23711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b93313a498d48eab540c057f32d828e. Latest consensus state: current_term: 1 leader_uuid: "2b93313a498d48eab540c057f32d828e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } } }
I20260927 07:03:31.882975 23712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b93313a498d48eab540c057f32d828e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b93313a498d48eab540c057f32d828e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41757 } } }
I20260927 07:03:31.883168 23712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:31.883116 23711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:31.883438 23720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:31.883985 23720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:31.885385 23720 catalog_manager.cc:1347] Generated new cluster ID: 377b3270f2ee47ba9c1a437447223c06
I20260927 07:03:31.885434 23720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:31.887072 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:31.881431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39169 (local address 127.0.0.1:41757)
0927 07:03:31.881653 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:31.881663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:31.881705 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:03:31.881829 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:31.881836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:31.881889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:31.882035 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:31.882253 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:31.883775 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:31.884990 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:31.885559 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:31.885672 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:31.886327 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:31.886331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:31.886331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:31.886523 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:31.886558 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:31.886572 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:31.886654 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:31.886840 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 07:03:31.886871 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260927 07:03:31.887960 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:31.892953 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:31.889336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52515 (local address 127.0.0.1:41757)
0927 07:03:31.889370 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:31.889372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:31.889578 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:03:31.889621 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:31.889622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:31.889631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:31.889689 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:31.889817 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:31.890546 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:31.891527 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:31.891962 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:31.892048 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:31.892698 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:31.892699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:31.892700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:31.892704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:31.892719 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:31.892723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:31.892797 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:31.892869 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 07:03:31.892882 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:31.900990 23720 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:31.901644 23720 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:31.908484 23720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b93313a498d48eab540c057f32d828e: Generated new TSK 0
I20260927 07:03:31.908633 23720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:31.911288 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23638
W20260927 07:03:31.915069 23623 connection.cc:535] client connection to 127.0.0.1:41757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41757): Connection reset by peer (error 104)
2026-09-27T07:03:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:31 dist-test-slave-mk3h krb5kdc[23737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:31 dist-test-slave-mk3h krb5kdc[23737](info): set up 2 sockets
Sep 27 07:03:31 dist-test-slave-mk3h krb5kdc[23737](info): commencing operation
krb5kdc: starting...
W20260927 07:03:32.935856 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:33 dist-test-slave-mk3h krb5kdc[23737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:33.046254 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:33.123586 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:33.123778 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:33.123801 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:33.124953 23750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:33.124995 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:33.125010 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:33.125022 23750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:33.125078 23750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:33.126497 23750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:33.126750 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:33.126983 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:33.131893 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:33.131989 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:33.132056 23750 server_base.cc:1034] running on GCE node
W20260927 07:03:33.131999 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:33.132464 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:33.132704 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:33.133836 23750 hybrid_clock.cc:648] HybridClock initialized: now 1790492613133824 us; error 33 us; skew 500 ppm
Sep 27 07:03:33 dist-test-slave-mk3h krb5kdc[23737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:33.143458 23750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:33.143936 23750 webserver.cc:460] Webserver started at http://127.0.0.1:42363/ using document root <none> and password file <none>
I20260927 07:03:33.144155 23750 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:33.144200 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:33.144354 23750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:33.145429 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "a0de055b2bc54eafbafe30580c087079"
format_stamp: "Formatted at 2026-09-27 07:03:33 on dist-test-slave-mk3h"
server_key: "c0069e9c52bc71aeef043fd3928a2c9e"
server_key_iv: "c469fc85ff72a0de69bb202cd15001dc"
server_key_version: "encryptionkey@0"
I20260927 07:03:33.145808 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "a0de055b2bc54eafbafe30580c087079"
format_stamp: "Formatted at 2026-09-27 07:03:33 on dist-test-slave-mk3h"
server_key: "c0069e9c52bc71aeef043fd3928a2c9e"
server_key_iv: "c469fc85ff72a0de69bb202cd15001dc"
server_key_version: "encryptionkey@0"
I20260927 07:03:33.147081 23750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:33.147866 23765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:33.148032 23750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 07:03:33.148131 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "a0de055b2bc54eafbafe30580c087079"
format_stamp: "Formatted at 2026-09-27 07:03:33 on dist-test-slave-mk3h"
server_key: "c0069e9c52bc71aeef043fd3928a2c9e"
server_key_iv: "c469fc85ff72a0de69bb202cd15001dc"
server_key_version: "encryptionkey@0"
I20260927 07:03:33.148200 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:33.151350 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:33.153126 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:33.153278 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:33.157094 23750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37377
I20260927 07:03:33.157125 23817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37377 every 8 connection(s)
I20260927 07:03:33.157437 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:33.158313 23818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:33.159981 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23750
I20260927 07:03:33.160099 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:33.160285 3300 external_mini_cluster.cc:1428] Setting key ea2cb4b678965b84c52e15f9b8a006b4
I20260927 07:03:33.160888 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079: Bootstrap starting.
I20260927 07:03:33.161489 23818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:33.161862 23818 log.cc:824] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:33.162650 23818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079: No bootstrap required, opened a new log
I20260927 07:03:33.162726 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:33.160750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35327 (local address 127.0.0.1:37377)
0927 07:03:33.160953 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:33.160960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:33.161006 (+ 46us) server_negotiation.cc:407] Connection header received
0927 07:03:33.161408 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:33.161413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:33.161442 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:33.161572 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:33.162227 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:33.162230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:33.162231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:33.162236 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:33.162299 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:33.162314 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:33.162384 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:33.162454 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 07:03:33.162499 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260927 07:03:33.163681 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:33.165211 23818 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260927 07:03:33.165393 23818 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:33.165447 23818 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0de055b2bc54eafbafe30580c087079, State: Initialized, Role: FOLLOWER
I20260927 07:03:33.165582 23818 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260927 07:03:33.165692 23818 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:33.165740 23818 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:33.165796 23818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:33.166270 23821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:33.165270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55751 (local address 127.0.0.1:37377)
0927 07:03:33.165314 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:33.165316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:33.165401 (+ 85us) server_negotiation.cc:407] Connection header received
0927 07:03:33.165481 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:33.165482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:33.165486 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:33.165545 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:33.166079 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:33.166080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:33.166081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:33.166084 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:33.166095 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:33.166098 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:33.166141 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:33.166184 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 07:03:33.166194 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 07:03:33.166498 23818 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260927 07:03:33.166602 23818 leader_election.cc:302] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0de055b2bc54eafbafe30580c087079; no voters:
I20260927 07:03:33.166766 23818 leader_election.cc:288] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:33.166786 23828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:33.166904 23828 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [term 1 LEADER]: Becoming Leader. State: Replica: a0de055b2bc54eafbafe30580c087079, State: Running, Role: LEADER
I20260927 07:03:33.167033 23818 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:33.167035 23828 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260927 07:03:33.167459 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0de055b2bc54eafbafe30580c087079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260927 07:03:33.167481 23830 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0de055b2bc54eafbafe30580c087079. Latest consensus state: current_term: 1 leader_uuid: "a0de055b2bc54eafbafe30580c087079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0de055b2bc54eafbafe30580c087079" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260927 07:03:33.167618 23830 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:33.167811 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:33.168145 23837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:33.168536 23837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:33.169986 23837 catalog_manager.cc:1347] Generated new cluster ID: 4a59fd9b204648988e84f1dcd9db0164
I20260927 07:03:33.170048 23837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:33.182531 23837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:33.182685 23837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:33.201737 23837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0de055b2bc54eafbafe30580c087079: Generated new TSK 0
I20260927 07:03:33.201943 23837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:33.204095 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23750
2026-09-27T07:03:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:33 dist-test-slave-mk3h krb5kdc[23849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:33 dist-test-slave-mk3h krb5kdc[23849](info): set up 2 sockets
Sep 27 07:03:33 dist-test-slave-mk3h krb5kdc[23849](info): commencing operation
krb5kdc: starting...
W20260927 07:03:34.219493 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:34.330704 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:34.409575 23862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:34.409788 23862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:34.409823 23862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:34.410912 23862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:34.410976 23862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:34.411000 23862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:34.411026 23862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:34.411049 23862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:34.412427 23862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:34.412664 23862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:34.412859 23862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:34.417307 23868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:34.417377 23862 server_base.cc:1034] running on GCE node
W20260927 07:03:34.417310 23867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:34.417522 23870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:34.417718 23862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:34.418022 23862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:34.419181 23862 hybrid_clock.cc:648] HybridClock initialized: now 1790492614419171 us; error 50 us; skew 500 ppm
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:34.421623 23862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:34.422047 23862 webserver.cc:460] Webserver started at http://127.0.0.1:42417/ using document root <none> and password file <none>
I20260927 07:03:34.422277 23862 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:34.422322 23862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:34.422464 23862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:34.423381 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "3443cff1872c4790ab045ec77b191823"
format_stamp: "Formatted at 2026-09-27 07:03:34 on dist-test-slave-mk3h"
server_key: "f1b99f3f78de552407df1d349a7c5535"
server_key_iv: "68a77f8310ae4dea8185510cff3ec971"
server_key_version: "encryptionkey@0"
I20260927 07:03:34.423727 23862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "3443cff1872c4790ab045ec77b191823"
format_stamp: "Formatted at 2026-09-27 07:03:34 on dist-test-slave-mk3h"
server_key: "f1b99f3f78de552407df1d349a7c5535"
server_key_iv: "68a77f8310ae4dea8185510cff3ec971"
server_key_version: "encryptionkey@0"
I20260927 07:03:34.424968 23862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:03:34.425769 23877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:34.425967 23862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:03:34.426045 23862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "3443cff1872c4790ab045ec77b191823"
format_stamp: "Formatted at 2026-09-27 07:03:34 on dist-test-slave-mk3h"
server_key: "f1b99f3f78de552407df1d349a7c5535"
server_key_iv: "68a77f8310ae4dea8185510cff3ec971"
server_key_version: "encryptionkey@0"
I20260927 07:03:34.426097 23862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:34.439348 23862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:34.441485 23862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:34.441598 23862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:34.445274 23862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43449
I20260927 07:03:34.445349 23929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43449 every 8 connection(s)
I20260927 07:03:34.445655 23862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:34.446617 23930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:34.448977 23930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823: Bootstrap starting.
I20260927 07:03:34.449529 23930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:34.449816 23930 log.cc:824] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:34.450482 23930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823: No bootstrap required, opened a new log
I20260927 07:03:34.452099 23930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260927 07:03:34.452255 23930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:34.452301 23930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3443cff1872c4790ab045ec77b191823, State: Initialized, Role: FOLLOWER
I20260927 07:03:34.452481 23930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260927 07:03:34.452579 23930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:34.452621 23930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:34.452674 23930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:34.453161 23930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260927 07:03:34.453276 23930 leader_election.cc:302] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3443cff1872c4790ab045ec77b191823; no voters:
I20260927 07:03:34.453454 23930 leader_election.cc:288] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:34.453477 23933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:34.453641 23933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [term 1 LEADER]: Becoming Leader. State: Replica: 3443cff1872c4790ab045ec77b191823, State: Running, Role: LEADER
I20260927 07:03:34.453697 23930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:34.453776 23933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260927 07:03:34.454119 23934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3443cff1872c4790ab045ec77b191823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260927 07:03:34.454222 23934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:34.454123 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3443cff1872c4790ab045ec77b191823. Latest consensus state: current_term: 1 leader_uuid: "3443cff1872c4790ab045ec77b191823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3443cff1872c4790ab045ec77b191823" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260927 07:03:34.454350 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:34.454468 23937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:34.454871 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23862
I20260927 07:03:34.454918 23937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:34.454998 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:34.455209 3300 external_mini_cluster.cc:1428] Setting key db93b51552f47f0e2df5371eb0567f1f
I20260927 07:03:34.456533 23937 catalog_manager.cc:1347] Generated new cluster ID: a67e842af5c54483978a423a5eb326e7
I20260927 07:03:34.456583 23937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:34.466323 23937 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:34.466972 23937 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:03:34.472458 23937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3443cff1872c4790ab045ec77b191823: Generated new TSK 0
I20260927 07:03:34.472607 23937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:34.473788 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:34.455641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41395 (local address 127.0.0.1:43449)
0927 07:03:34.456210 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:34.456218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:34.456248 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:03:34.456445 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:34.456451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:34.463650 (+ 7199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:34.463802 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:34.464543 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:34.466054 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:34.467107 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:34.467721 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:34.467834 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:34.470529 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:34.470532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:34.470533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:34.470544 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:34.472385 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:34.472778 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:34.472781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:34.472781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:34.472868 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:34.473064 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:34.473064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:34.473064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:34.473242 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:34.473349 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:34.473453 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 07:03:34.473547 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":68,"threads_started":1}
I20260927 07:03:34.474575 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:34.482162 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:34.475944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60757 (local address 127.0.0.1:43449)
0927 07:03:34.475980 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:34.475982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:34.476062 (+ 80us) server_negotiation.cc:407] Connection header received
0927 07:03:34.476127 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:34.476127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:34.476142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:34.476184 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:34.476797 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:34.477250 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:34.478308 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:34.478499 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:34.478542 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:34.479825 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:34.479827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:34.479827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:34.479832 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:34.481195 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:34.481555 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:34.481556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:34.481556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:34.481600 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:34.481786 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:34.481786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:34.481786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:34.481869 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:34.481946 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:34.482060 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 07:03:34.482112 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:34.484136 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23862
2026-09-27T07:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23961](info): set up 2 sockets
Sep 27 07:03:34 dist-test-slave-mk3h krb5kdc[23961](info): commencing operation
krb5kdc: starting...
W20260927 07:03:35.507372 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:35 dist-test-slave-mk3h krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:35.618639 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:35.696444 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:35.696656 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:35.696694 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:35.697821 23974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:35.697885 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:35.697917 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:35.697948 23974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:35.697973 23974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:35.699429 23974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:35.699712 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:35.699930 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:35.704558 23980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:35.704591 23979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:35.704655 23974 server_base.cc:1034] running on GCE node
W20260927 07:03:35.704558 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:35.704993 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:35.705268 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:35.706399 23974 hybrid_clock.cc:648] HybridClock initialized: now 1790492615706379 us; error 34 us; skew 500 ppm
Sep 27 07:03:35 dist-test-slave-mk3h krb5kdc[23961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:35.708808 23974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:35.709225 23974 webserver.cc:460] Webserver started at http://127.0.0.1:46131/ using document root <none> and password file <none>
I20260927 07:03:35.709450 23974 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:35.709494 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:35.709653 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:35.710647 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "c660e5e3549941f5b88c338169a3859d"
format_stamp: "Formatted at 2026-09-27 07:03:35 on dist-test-slave-mk3h"
server_key: "7ef115438b0e6d805e98af540008f310"
server_key_iv: "8f34851eb9f2371e7d3529ee12e48537"
server_key_version: "encryptionkey@0"
I20260927 07:03:35.710999 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "c660e5e3549941f5b88c338169a3859d"
format_stamp: "Formatted at 2026-09-27 07:03:35 on dist-test-slave-mk3h"
server_key: "7ef115438b0e6d805e98af540008f310"
server_key_iv: "8f34851eb9f2371e7d3529ee12e48537"
server_key_version: "encryptionkey@0"
I20260927 07:03:35.712334 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:35.713008 23989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:35.713198 23974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:35.713286 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "c660e5e3549941f5b88c338169a3859d"
format_stamp: "Formatted at 2026-09-27 07:03:35 on dist-test-slave-mk3h"
server_key: "7ef115438b0e6d805e98af540008f310"
server_key_iv: "8f34851eb9f2371e7d3529ee12e48537"
server_key_version: "encryptionkey@0"
I20260927 07:03:35.713338 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:35.728178 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:35.730324 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:35.730423 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:35.733999 23974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45347
I20260927 07:03:35.734048 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45347 every 8 connection(s)
I20260927 07:03:35.734421 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:35.735811 24042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:35.739259 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d: Bootstrap starting.
I20260927 07:03:35.739921 24042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:35.740248 24042 log.cc:824] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:35.741044 24042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d: No bootstrap required, opened a new log
I20260927 07:03:35.742941 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 23974
I20260927 07:03:35.743052 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:35.743211 3300 external_mini_cluster.cc:1428] Setting key 54db3f69a12447aa74b2857e2a22d93a
I20260927 07:03:35.743444 24042 raft_consensus.cc:348] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260927 07:03:35.743614 24042 raft_consensus.cc:374] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:35.743646 24042 raft_consensus.cc:729] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c660e5e3549941f5b88c338169a3859d, State: Initialized, Role: FOLLOWER
I20260927 07:03:35.744022 24042 consensus_queue.cc:260] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260927 07:03:35.744115 24042 raft_consensus.cc:388] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:35.744146 24042 raft_consensus.cc:482] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:35.744191 24042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:35.745260 24042 raft_consensus.cc:504] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260927 07:03:35.745364 24042 leader_election.cc:302] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c660e5e3549941f5b88c338169a3859d; no voters:
I20260927 07:03:35.745512 24042 leader_election.cc:288] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:35.745572 24047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:35.745726 24042 sys_catalog.cc:564] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:35.745743 24047 raft_consensus.cc:686] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [term 1 LEADER]: Becoming Leader. State: Replica: c660e5e3549941f5b88c338169a3859d, State: Running, Role: LEADER
I20260927 07:03:35.745828 24047 consensus_queue.cc:237] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } }
I20260927 07:03:35.746196 24048 sys_catalog.cc:455] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c660e5e3549941f5b88c338169a3859d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } } }
I20260927 07:03:35.746307 24048 sys_catalog.cc:458] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:35.746171 24049 sys_catalog.cc:455] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c660e5e3549941f5b88c338169a3859d. Latest consensus state: current_term: 1 leader_uuid: "c660e5e3549941f5b88c338169a3859d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c660e5e3549941f5b88c338169a3859d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45347 } } }
I20260927 07:03:35.746560 24049 sys_catalog.cc:458] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:35.746559 24056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:35.747211 24056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:35.748885 24056 catalog_manager.cc:1347] Generated new cluster ID: b06a2609a72e47159572359e0052e07e
I20260927 07:03:35.748939 24056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:35.749387 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:35.743855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56747 (local address 127.0.0.1:45347)
0927 07:03:35.744022 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:35.744030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:35.744059 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:03:35.744163 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:35.744170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:35.744222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:35.744361 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:35.744545 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:35.745988 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:35.747153 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:35.747773 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:35.747878 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:35.748665 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:35.748668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:35.748668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:35.748898 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:35.748953 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:35.748968 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:35.749041 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:35.749156 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 07:03:35.749218 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260927 07:03:35.751961 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:35.754354 24056 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:35.754802 24056 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:35.757452 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:35.753288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59623 (local address 127.0.0.1:45347)
0927 07:03:35.753324 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:35.753326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:35.754114 (+ 788us) server_negotiation.cc:407] Connection header received
0927 07:03:35.754161 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:35.754163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:35.754175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:35.754204 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:35.754387 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:35.755071 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:35.756139 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:35.756411 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:35.756450 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:35.757181 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:35.757182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:35.757182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:35.757186 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:35.757199 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:35.757202 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:35.757229 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:35.757335 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:03:35.757375 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":19}
I20260927 07:03:35.758631 24056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c660e5e3549941f5b88c338169a3859d: Generated new TSK 0
I20260927 07:03:35.758797 24056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:35.765164 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 23974
2026-09-27T07:03:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:35 dist-test-slave-mk3h krb5kdc[24073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:35 dist-test-slave-mk3h krb5kdc[24073](info): set up 2 sockets
Sep 27 07:03:35 dist-test-slave-mk3h krb5kdc[24073](info): commencing operation
krb5kdc: starting...
W20260927 07:03:36.795657 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:36 dist-test-slave-mk3h krb5kdc[24073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:36.905607 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:36.982295 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:36.982508 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:36.982546 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:36.983744 24086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:36.983820 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:36.983845 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:36.983877 24086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:36.983902 24086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:36.985356 24086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:36.985627 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:36.985852 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:36.990397 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:36.990437 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:36.990432 24086 server_base.cc:1034] running on GCE node
W20260927 07:03:36.990430 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:36.990787 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:36.991014 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:36.992146 24086 hybrid_clock.cc:648] HybridClock initialized: now 1790492616992125 us; error 34 us; skew 500 ppm
Sep 27 07:03:36 dist-test-slave-mk3h krb5kdc[24073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:36.994881 24086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:36.995352 24086 webserver.cc:460] Webserver started at http://127.0.0.1:33819/ using document root <none> and password file <none>
I20260927 07:03:36.995611 24086 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:36.995684 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:36.995817 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:36.996803 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "4936324497784199a53995916100a41f"
format_stamp: "Formatted at 2026-09-27 07:03:36 on dist-test-slave-mk3h"
server_key: "4da5fde54baab12e87379ed1a5117ead"
server_key_iv: "d3d266340d9eec1b87a573df730d280e"
server_key_version: "encryptionkey@0"
I20260927 07:03:36.997153 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "4936324497784199a53995916100a41f"
format_stamp: "Formatted at 2026-09-27 07:03:36 on dist-test-slave-mk3h"
server_key: "4da5fde54baab12e87379ed1a5117ead"
server_key_iv: "d3d266340d9eec1b87a573df730d280e"
server_key_version: "encryptionkey@0"
I20260927 07:03:36.998366 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:36.999015 24101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:36.999194 24086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:36.999301 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "4936324497784199a53995916100a41f"
format_stamp: "Formatted at 2026-09-27 07:03:36 on dist-test-slave-mk3h"
server_key: "4da5fde54baab12e87379ed1a5117ead"
server_key_iv: "d3d266340d9eec1b87a573df730d280e"
server_key_version: "encryptionkey@0"
I20260927 07:03:36.999382 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:37.002486 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:37.004312 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:37.004462 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:37.008267 24086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40861
I20260927 07:03:37.008296 24153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40861 every 8 connection(s)
I20260927 07:03:37.008597 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:37.009415 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24086
I20260927 07:03:37.009541 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:37.009577 24154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:37.009744 3300 external_mini_cluster.cc:1428] Setting key 678fd7cf61809b04ad1db4fb8f3b5487
I20260927 07:03:37.012216 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:37.010180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49475 (local address 127.0.0.1:40861)
0927 07:03:37.010375 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:37.010384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:37.010416 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:03:37.011091 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:37.011099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:37.011131 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:37.011281 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:37.011835 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:37.011838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:37.011838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:37.011843 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:37.011882 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:37.011895 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:37.011971 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:37.012006 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 07:03:37.012042 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260927 07:03:37.012923 24154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f: Bootstrap starting.
I20260927 07:03:37.013152 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:37.013602 24154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:37.013923 24154 log.cc:824] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:37.014582 24154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f: No bootstrap required, opened a new log
I20260927 07:03:37.015604 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:37.014700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48487 (local address 127.0.0.1:40861)
0927 07:03:37.014747 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:37.014749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:37.014842 (+ 93us) server_negotiation.cc:407] Connection header received
0927 07:03:37.014894 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:37.014895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:37.014900 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:37.014956 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:37.015396 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:37.015398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:37.015398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:37.015402 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:37.015414 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:37.015418 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:37.015475 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:37.015507 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 07:03:37.015518 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 07:03:37.016806 24154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 07:03:37.016942 24154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:37.016963 24154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4936324497784199a53995916100a41f, State: Initialized, Role: FOLLOWER
I20260927 07:03:37.017113 24154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 07:03:37.017185 24154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:37.017223 24154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:37.017270 24154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:37.017745 24154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 07:03:37.017844 24154 leader_election.cc:302] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4936324497784199a53995916100a41f; no voters:
I20260927 07:03:37.018036 24154 leader_election.cc:288] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:37.018066 24164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:37.018260 24154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:37.018362 24164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [term 1 LEADER]: Becoming Leader. State: Replica: 4936324497784199a53995916100a41f, State: Running, Role: LEADER
I20260927 07:03:37.018499 24164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } }
I20260927 07:03:37.018901 24166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4936324497784199a53995916100a41f. Latest consensus state: current_term: 1 leader_uuid: "4936324497784199a53995916100a41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } } }
I20260927 07:03:37.018922 24165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4936324497784199a53995916100a41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4936324497784199a53995916100a41f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40861 } } }
I20260927 07:03:37.018988 24166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:37.019011 24165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:37.019199 24173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:37.019672 24173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:37.021238 24173 catalog_manager.cc:1347] Generated new cluster ID: 36575bdeef10481a98eadfe7a8e07b39
I20260927 07:03:37.021287 24173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:37.033232 24173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:37.033411 24173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:37.040421 24173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4936324497784199a53995916100a41f: Generated new TSK 0
I20260927 07:03:37.040622 24173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:37.047029 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24086
2026-09-27T07:03:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:37 dist-test-slave-mk3h krb5kdc[24185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:37 dist-test-slave-mk3h krb5kdc[24185](info): set up 2 sockets
Sep 27 07:03:37 dist-test-slave-mk3h krb5kdc[24185](info): commencing operation
krb5kdc: starting...
W20260927 07:03:38.103518 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:38.215219 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:38.293880 24198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:38.294067 24198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:38.294090 24198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:38.295255 24198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:38.295295 24198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:38.295310 24198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:38.295351 24198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:38.295380 24198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:38.296854 24198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:38.297109 24198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:38.297322 24198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:38.302089 24206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:38.302248 24198 server_base.cc:1034] running on GCE node
W20260927 07:03:38.302119 24204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:38.302089 24203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:38.302570 24198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:38.302876 24198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:38.304036 24198 hybrid_clock.cc:648] HybridClock initialized: now 1790492618304024 us; error 45 us; skew 500 ppm
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:38.312016 24198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:38.312448 24198 webserver.cc:460] Webserver started at http://127.0.0.1:44431/ using document root <none> and password file <none>
I20260927 07:03:38.312673 24198 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:38.312742 24198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:38.312902 24198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:38.313922 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "28511192095441939b0b8fa8a2385163"
format_stamp: "Formatted at 2026-09-27 07:03:38 on dist-test-slave-mk3h"
server_key: "074e43abd3a4b9a9c6fd44b96159e048"
server_key_iv: "66f7f3bb958ad72321b402fe337667c1"
server_key_version: "encryptionkey@0"
I20260927 07:03:38.314285 24198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "28511192095441939b0b8fa8a2385163"
format_stamp: "Formatted at 2026-09-27 07:03:38 on dist-test-slave-mk3h"
server_key: "074e43abd3a4b9a9c6fd44b96159e048"
server_key_iv: "66f7f3bb958ad72321b402fe337667c1"
server_key_version: "encryptionkey@0"
I20260927 07:03:38.315600 24198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:03:38.316346 24213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:38.316517 24198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:38.316620 24198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "28511192095441939b0b8fa8a2385163"
format_stamp: "Formatted at 2026-09-27 07:03:38 on dist-test-slave-mk3h"
server_key: "074e43abd3a4b9a9c6fd44b96159e048"
server_key_iv: "66f7f3bb958ad72321b402fe337667c1"
server_key_version: "encryptionkey@0"
I20260927 07:03:38.316709 24198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:38.336356 24198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:38.338217 24198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:38.338375 24198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:38.342325 24198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41501
I20260927 07:03:38.342356 24265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41501 every 8 connection(s)
I20260927 07:03:38.342674 24198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:38.343616 24266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:38.345764 24266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: Bootstrap starting.
I20260927 07:03:38.346328 24266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:38.346603 24266 log.cc:824] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:38.347263 24266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: No bootstrap required, opened a new log
I20260927 07:03:38.348876 24266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } }
I20260927 07:03:38.349020 24266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:38.349044 24266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28511192095441939b0b8fa8a2385163, State: Initialized, Role: FOLLOWER
I20260927 07:03:38.349221 24266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } }
I20260927 07:03:38.349284 24266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:38.349305 24266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:38.349371 24266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:38.349570 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24198
I20260927 07:03:38.349691 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:38.349887 3300 external_mini_cluster.cc:1428] Setting key 2d646981f98e9383ecd76e934b73ca62
I20260927 07:03:38.350077 24266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } }
I20260927 07:03:38.350196 24266 leader_election.cc:302] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28511192095441939b0b8fa8a2385163; no voters:
I20260927 07:03:38.350390 24266 leader_election.cc:288] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:38.350428 24270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:38.350584 24270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [term 1 LEADER]: Becoming Leader. State: Replica: 28511192095441939b0b8fa8a2385163, State: Running, Role: LEADER
I20260927 07:03:38.350629 24266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:38.351501 24270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } }
I20260927 07:03:38.352155 24272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28511192095441939b0b8fa8a2385163. Latest consensus state: current_term: 1 leader_uuid: "28511192095441939b0b8fa8a2385163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } } }
I20260927 07:03:38.352324 24273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28511192095441939b0b8fa8a2385163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28511192095441939b0b8fa8a2385163" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41501 } } }
I20260927 07:03:38.352445 24272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:38.352451 24273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163 [sys.catalog]: This master's current role is: LEADER
W20260927 07:03:38.352849 24285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:38.352921 24285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:38.353003 24287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:38.353160 24287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:38.354689 24287 catalog_manager.cc:1347] Generated new cluster ID: 842e63efe3a84f48bda7a05335eeca17
I20260927 07:03:38.354743 24287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:03:38.361855 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:38.350294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:60797 (local address 127.0.0.1:41501)
0927 07:03:38.350576 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:38.350585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:38.350618 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:03:38.350747 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:38.350755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:38.350816 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:38.350965 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:38.351836 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:38.353836 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:38.354877 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:38.355271 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:38.355391 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:38.358052 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:38.358056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:38.358056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:38.358067 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:38.360260 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:38.360656 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:38.360659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:38.360659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:38.360765 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:38.361002 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:38.361003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:38.361004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:38.361257 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:38.361429 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:38.361539 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 07:03:38.361643 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":70,"threads_started":1}
I20260927 07:03:38.362752 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:38.369606 24287 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:38.370291 24287 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:38.370371 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:38.364102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:44919 (local address 127.0.0.1:41501)
0927 07:03:38.364141 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:38.364143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:38.364263 (+ 120us) server_negotiation.cc:407] Connection header received
0927 07:03:38.364336 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:38.364337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:38.364347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:38.364399 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:38.364898 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:38.365365 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:38.366063 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:38.366301 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:38.366390 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:38.367494 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:38.367495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:38.367495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:38.367502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:38.368937 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:38.369405 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:38.369405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:38.369406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:38.369485 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:38.369745 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:38.369746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:38.369746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:38.369855 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:38.369985 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:38.370088 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 07:03:38.370222 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":18}
I20260927 07:03:38.380502 24287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28511192095441939b0b8fa8a2385163: Generated new TSK 0
I20260927 07:03:38.380667 24287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:38.385761 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24198
W20260927 07:03:38.390187 24183 connection.cc:535] client connection to 127.0.0.1:41501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41501): Connection reset by peer (error 104)
2026-09-27T07:03:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24297](info): set up 2 sockets
Sep 27 07:03:38 dist-test-slave-mk3h krb5kdc[24297](info): commencing operation
krb5kdc: starting...
W20260927 07:03:39.399447 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:39 dist-test-slave-mk3h krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:39.519294 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:39.598126 24310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:39.598341 24310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:39.598379 24310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:39.599505 24310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:39.599587 24310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:39.599613 24310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:39.599641 24310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:39.599665 24310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:39.601040 24310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:39.601280 24310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:39.601473 24310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:39.606488 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:39.606568 24318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:39.606480 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:39.607146 24310 server_base.cc:1034] running on GCE node
I20260927 07:03:39.607383 24310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:39.607720 24310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:39.608878 24310 hybrid_clock.cc:648] HybridClock initialized: now 1790492619608865 us; error 28 us; skew 500 ppm
Sep 27 07:03:39 dist-test-slave-mk3h krb5kdc[24297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:39.611193 24310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:39.611694 24310 webserver.cc:460] Webserver started at http://127.0.0.1:33891/ using document root <none> and password file <none>
I20260927 07:03:39.611934 24310 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:39.611999 24310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:39.612129 24310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:39.613113 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "7bd059c3be954e388c2306ca9b648bd7"
format_stamp: "Formatted at 2026-09-27 07:03:39 on dist-test-slave-mk3h"
server_key: "5dde41adf3bc11548772a147e64c69af"
server_key_iv: "6a69a2715254ba1d96d664ee54206ff7"
server_key_version: "encryptionkey@0"
I20260927 07:03:39.613476 24310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "7bd059c3be954e388c2306ca9b648bd7"
format_stamp: "Formatted at 2026-09-27 07:03:39 on dist-test-slave-mk3h"
server_key: "5dde41adf3bc11548772a147e64c69af"
server_key_iv: "6a69a2715254ba1d96d664ee54206ff7"
server_key_version: "encryptionkey@0"
I20260927 07:03:39.614729 24310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 07:03:39.615378 24325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:39.615594 24310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:39.615712 24310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "7bd059c3be954e388c2306ca9b648bd7"
format_stamp: "Formatted at 2026-09-27 07:03:39 on dist-test-slave-mk3h"
server_key: "5dde41adf3bc11548772a147e64c69af"
server_key_iv: "6a69a2715254ba1d96d664ee54206ff7"
server_key_version: "encryptionkey@0"
I20260927 07:03:39.615806 24310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:39.624594 24310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:39.626192 24310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:39.626340 24310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:39.630115 24310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46029
I20260927 07:03:39.630153 24377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46029 every 8 connection(s)
I20260927 07:03:39.630483 24310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:39.631338 24378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:39.633603 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24310
I20260927 07:03:39.633730 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:39.633940 3300 external_mini_cluster.cc:1428] Setting key 77f46b87d9963b7ead588b6dcc664385
I20260927 07:03:39.634168 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7: Bootstrap starting.
I20260927 07:03:39.634778 24378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:39.635035 24378 log.cc:824] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:39.635746 24378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7: No bootstrap required, opened a new log
I20260927 07:03:39.637457 24378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20260927 07:03:39.637575 24378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:39.637596 24378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd059c3be954e388c2306ca9b648bd7, State: Initialized, Role: FOLLOWER
I20260927 07:03:39.637719 24378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20260927 07:03:39.637789 24378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:39.637810 24378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:39.637876 24378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:39.638501 24378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20260927 07:03:39.638626 24378 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd059c3be954e388c2306ca9b648bd7; no voters:
I20260927 07:03:39.638783 24378 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:39.638813 24383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:39.638991 24383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd059c3be954e388c2306ca9b648bd7, State: Running, Role: LEADER
I20260927 07:03:39.639019 24378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:39.639130 24383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20260927 07:03:39.639408 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:39.634357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:39103 (local address 127.0.0.1:46029)
0927 07:03:39.634516 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:39.634526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:39.634570 (+ 44us) server_negotiation.cc:407] Connection header received
0927 07:03:39.635029 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:39.635036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:39.635105 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:39.635248 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:39.635478 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:39.636547 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:39.637334 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:39.637856 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:39.637957 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:39.638616 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:39.638620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:39.638620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:39.638823 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:39.638858 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:39.638873 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:39.639167 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:39.639209 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 07:03:39.639236 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 07:03:39.639472 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd059c3be954e388c2306ca9b648bd7. Latest consensus state: current_term: 1 leader_uuid: "7bd059c3be954e388c2306ca9b648bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } } }
I20260927 07:03:39.639664 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:39.639603 24384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd059c3be954e388c2306ca9b648bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd059c3be954e388c2306ca9b648bd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } } }
I20260927 07:03:39.639906 24384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:39.640275 24392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:39.640376 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:39.640739 24392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:39.644003 24392 catalog_manager.cc:1347] Generated new cluster ID: db2cebc97a8a4256a64a41292a1518ca
I20260927 07:03:39.644066 24392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:39.644659 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:39.641693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:43921 (local address 127.0.0.1:46029)
0927 07:03:39.641725 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:39.641727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:39.641827 (+ 100us) server_negotiation.cc:407] Connection header received
0927 07:03:39.641906 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:39.641907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:39.641914 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:39.641971 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:39.642081 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:39.642498 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:39.643450 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:39.643707 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:39.643781 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:39.644434 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:39.644436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:39.644436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:39.644440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:39.644453 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:39.644457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:39.644518 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:39.644584 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:03:39.644593 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:03:39.655301 24392 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:39.655844 24392 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:39.659401 24392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd059c3be954e388c2306ca9b648bd7: Generated new TSK 0
I20260927 07:03:39.659593 24392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:39.666347 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24310
W20260927 07:03:39.670516 24295 connection.cc:535] client connection to 127.0.0.1:46029 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46029): Connection reset by peer (error 104)
2026-09-27T07:03:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:39 dist-test-slave-mk3h krb5kdc[24409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:39 dist-test-slave-mk3h krb5kdc[24409](info): set up 2 sockets
Sep 27 07:03:39 dist-test-slave-mk3h krb5kdc[24409](info): commencing operation
krb5kdc: starting...
W20260927 07:03:40.708532 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:40 dist-test-slave-mk3h krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:40.860594 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:40.952720 24422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:40.952963 24422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:40.952989 24422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:40.954524 24422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:40.954571 24422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:40.954586 24422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:40.954598 24422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:40.954610 24422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:40.956270 24422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:40.956537 24422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:40.956801 24422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:40.962778 24427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:40.963529 24430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:40.963598 24428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:40.964731 24422 server_base.cc:1034] running on GCE node
I20260927 07:03:40.965065 24422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:40.965411 24422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:40.967252 24422 hybrid_clock.cc:648] HybridClock initialized: now 1790492620967189 us; error 65 us; skew 500 ppm
Sep 27 07:03:40 dist-test-slave-mk3h krb5kdc[24409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:40.970364 24422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:40.970990 24422 webserver.cc:460] Webserver started at http://127.0.0.1:35037/ using document root <none> and password file <none>
I20260927 07:03:40.971225 24422 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:40.971271 24422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:40.971361 24422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:40.972507 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "d1d39ee3ebdd4ee09ca54218bf00a428"
format_stamp: "Formatted at 2026-09-27 07:03:40 on dist-test-slave-mk3h"
server_key: "c1da4b8479074e1e7b4c904afe12b8a8"
server_key_iv: "99e4496ecbad8817878b62e6555e2564"
server_key_version: "encryptionkey@0"
I20260927 07:03:40.972807 24422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "d1d39ee3ebdd4ee09ca54218bf00a428"
format_stamp: "Formatted at 2026-09-27 07:03:40 on dist-test-slave-mk3h"
server_key: "c1da4b8479074e1e7b4c904afe12b8a8"
server_key_iv: "99e4496ecbad8817878b62e6555e2564"
server_key_version: "encryptionkey@0"
I20260927 07:03:40.974143 24422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:40.975063 24437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:40.975381 24422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:40.975473 24422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "d1d39ee3ebdd4ee09ca54218bf00a428"
format_stamp: "Formatted at 2026-09-27 07:03:40 on dist-test-slave-mk3h"
server_key: "c1da4b8479074e1e7b4c904afe12b8a8"
server_key_iv: "99e4496ecbad8817878b62e6555e2564"
server_key_version: "encryptionkey@0"
I20260927 07:03:40.975713 24422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:40.980175 24422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:40.982487 24422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:40.982682 24422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:40.986732 24422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43223
I20260927 07:03:40.986761 24489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43223 every 8 connection(s)
I20260927 07:03:40.987103 24422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:40.988143 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24422
I20260927 07:03:40.988204 24490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:40.988286 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:40.988480 3300 external_mini_cluster.cc:1428] Setting key ebf061ae532d64345166ba60d4389282
I20260927 07:03:40.990926 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:40.989007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:42979 (local address 127.0.0.1:43223)
0927 07:03:40.989196 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:40.989203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:40.989229 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:03:40.989705 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:40.989712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:40.989736 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:40.989843 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:40.990496 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:40.990499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:40.990499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:40.990503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:40.990538 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:40.990548 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:40.990603 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:40.990682 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 07:03:40.990736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260927 07:03:40.991606 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428: Bootstrap starting.
I20260927 07:03:40.991923 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:40.992343 24490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:40.992662 24490 log.cc:824] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:40.993376 24490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428: No bootstrap required, opened a new log
I20260927 07:03:40.994554 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:40.993604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:33229 (local address 127.0.0.1:43223)
0927 07:03:40.993645 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:40.993647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:40.993732 (+ 85us) server_negotiation.cc:407] Connection header received
0927 07:03:40.993796 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:40.993797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:40.993800 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:40.993842 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:40.994339 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:40.994340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:40.994340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:40.994344 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:40.994356 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:40.994359 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:40.994403 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:40.994455 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 07:03:40.994465 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:03:40.995868 24490 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } }
I20260927 07:03:40.996018 24490 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:40.996062 24490 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d39ee3ebdd4ee09ca54218bf00a428, State: Initialized, Role: FOLLOWER
I20260927 07:03:40.996187 24490 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } }
I20260927 07:03:40.996282 24490 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:40.996322 24490 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:40.996388 24490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:40.996917 24490 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } }
I20260927 07:03:40.997023 24490 leader_election.cc:302] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d39ee3ebdd4ee09ca54218bf00a428; no voters:
I20260927 07:03:40.997219 24490 leader_election.cc:288] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:40.997255 24500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:40.997411 24500 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [term 1 LEADER]: Becoming Leader. State: Replica: d1d39ee3ebdd4ee09ca54218bf00a428, State: Running, Role: LEADER
I20260927 07:03:40.997485 24490 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:40.997557 24500 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } }
I20260927 07:03:40.997939 24502 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d39ee3ebdd4ee09ca54218bf00a428. Latest consensus state: current_term: 1 leader_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } } }
I20260927 07:03:40.997928 24501 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d39ee3ebdd4ee09ca54218bf00a428" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43223 } } }
I20260927 07:03:40.998039 24502 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:40.998046 24501 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:40.998288 24508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:40.998782 24508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:41.000319 24508 catalog_manager.cc:1347] Generated new cluster ID: f05efcf628444cde8e60696c286fd497
I20260927 07:03:41.000381 24508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:41.018977 24508 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:41.019181 24508 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:41.029578 24508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d39ee3ebdd4ee09ca54218bf00a428: Generated new TSK 0
I20260927 07:03:41.029763 24508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:41.040170 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24422
2026-09-27T07:03:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:41 dist-test-slave-mk3h krb5kdc[24521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:41 dist-test-slave-mk3h krb5kdc[24521](info): set up 2 sockets
Sep 27 07:03:41 dist-test-slave-mk3h krb5kdc[24521](info): commencing operation
krb5kdc: starting...
W20260927 07:03:42.051724 3300 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:42.160213 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:42.237372 24534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:42.237548 24534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:42.237569 24534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:42.238673 24534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:42.238713 24534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:42.238727 24534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:42.238762 24534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:42.238791 24534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:42.240263 24534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:42.240499 24534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:42.240715 24534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:42.245477 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:42.245596 24539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:42.245464 24534 server_base.cc:1034] running on GCE node
W20260927 07:03:42.245468 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:42.245960 24534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:42.246186 24534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:42.247318 24534 hybrid_clock.cc:648] HybridClock initialized: now 1790492622247308 us; error 26 us; skew 500 ppm
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:42.249626 24534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:42.250056 24534 webserver.cc:460] Webserver started at http://127.0.0.1:43283/ using document root <none> and password file <none>
I20260927 07:03:42.250283 24534 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:42.250326 24534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:42.250461 24534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:42.251418 24534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "f4ae1dd856d74809a53f915f49a06134"
format_stamp: "Formatted at 2026-09-27 07:03:42 on dist-test-slave-mk3h"
server_key: "330f8d058226b29deed4d587a3eb9515"
server_key_iv: "771cdba11218721b2e1e27cbf66cc673"
server_key_version: "encryptionkey@0"
I20260927 07:03:42.251791 24534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "f4ae1dd856d74809a53f915f49a06134"
format_stamp: "Formatted at 2026-09-27 07:03:42 on dist-test-slave-mk3h"
server_key: "330f8d058226b29deed4d587a3eb9515"
server_key_iv: "771cdba11218721b2e1e27cbf66cc673"
server_key_version: "encryptionkey@0"
I20260927 07:03:42.253001 24534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:03:42.253630 24549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:42.253845 24534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:42.253960 24534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "f4ae1dd856d74809a53f915f49a06134"
format_stamp: "Formatted at 2026-09-27 07:03:42 on dist-test-slave-mk3h"
server_key: "330f8d058226b29deed4d587a3eb9515"
server_key_iv: "771cdba11218721b2e1e27cbf66cc673"
server_key_version: "encryptionkey@0"
I20260927 07:03:42.254031 24534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:42.266403 24534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:42.268224 24534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:42.268371 24534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:42.272281 24534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35361
I20260927 07:03:42.272308 24601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35361 every 8 connection(s)
I20260927 07:03:42.272648 24534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:42.273532 24602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:42.274163 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24534
I20260927 07:03:42.274291 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:42.274489 3300 external_mini_cluster.cc:1428] Setting key 1925a72fa80c98b7c4feffad89c1bf3f
I20260927 07:03:42.276412 24602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134: Bootstrap starting.
I20260927 07:03:42.277084 24602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:42.277371 24602 log.cc:824] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:42.278028 24602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134: No bootstrap required, opened a new log
I20260927 07:03:42.280131 24602 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260927 07:03:42.280251 24602 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:42.280272 24602 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ae1dd856d74809a53f915f49a06134, State: Initialized, Role: FOLLOWER
I20260927 07:03:42.280378 24602 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260927 07:03:42.280448 24602 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:42.280469 24602 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:42.280530 24602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:42.281142 24602 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260927 07:03:42.281248 24602 leader_election.cc:302] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ae1dd856d74809a53f915f49a06134; no voters:
I20260927 07:03:42.281395 24602 leader_election.cc:288] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:03:42.281456 24607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:42.281627 24607 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [term 1 LEADER]: Becoming Leader. State: Replica: f4ae1dd856d74809a53f915f49a06134, State: Running, Role: LEADER
I20260927 07:03:42.281625 24602 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:42.281734 24607 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } }
I20260927 07:03:42.282032 24607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ae1dd856d74809a53f915f49a06134. Latest consensus state: current_term: 1 leader_uuid: "f4ae1dd856d74809a53f915f49a06134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } } }
I20260927 07:03:42.282172 24607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:42.282356 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ae1dd856d74809a53f915f49a06134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ae1dd856d74809a53f915f49a06134" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35361 } } }
I20260927 07:03:42.282430 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:42.282506 24616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:42.282960 24616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:42.284832 24616 catalog_manager.cc:1347] Generated new cluster ID: 53b0a439ec95411cbf4501153cf17c53
I20260927 07:03:42.284888 24616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:42.285779 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:42.274974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:50671 (local address 127.0.0.1:35361)
0927 07:03:42.275158 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:42.275166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:42.275196 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:03:42.275743 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:42.275751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:42.275812 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:42.275940 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:42.276506 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:42.277822 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:42.278614 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:42.279221 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:42.279328 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:42.281950 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:42.281954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:42.281955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:42.281968 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:42.284204 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:42.284622 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:42.284624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:42.284624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:42.284714 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:42.284985 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:42.284986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:42.284987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:42.285250 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:42.285371 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:42.285523 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:03:42.285607 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260927 07:03:42.286669 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:42.295068 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:42.288133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:59841 (local address 127.0.0.1:35361)
0927 07:03:42.288166 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:42.288168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:42.288320 (+ 152us) server_negotiation.cc:407] Connection header received
0927 07:03:42.288389 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:42.288390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:42.288399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:42.288447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:42.289117 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:42.289795 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:42.290775 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:42.291067 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:42.291124 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:42.292408 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:42.292409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:42.292410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:42.292416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:42.293913 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:42.294311 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:42.294313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:42.294313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:42.294371 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:42.294582 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:42.294583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:42.294605 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:42.294727 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:42.294819 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:42.294926 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:03:42.294983 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:03:42.297801 24616 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:42.298237 24616 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:42.302253 24616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ae1dd856d74809a53f915f49a06134: Generated new TSK 0
I20260927 07:03:42.302397 24616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:42.305971 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24534
W20260927 07:03:42.310273 24519 connection.cc:535] client connection to 127.0.0.1:35361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35361): Connection reset by peer (error 104)
2026-09-27T07:03:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24633](info): set up 2 sockets
Sep 27 07:03:42 dist-test-slave-mk3h krb5kdc[24633](info): commencing operation
krb5kdc: starting...
W20260927 07:03:43.327661 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:43 dist-test-slave-mk3h krb5kdc[24633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:43.438460 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:43.514979 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:43.515158 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:43.515179 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:43.516258 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:43.516294 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:43.516306 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:43.516319 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:43.516330 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:43.517642 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:43.517800 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:43.517941 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:03:43.522486 24646 server_base.cc:1034] running on GCE node
W20260927 07:03:43.522470 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:43.522468 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:43.522441 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:43.522967 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:43.523234 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:43.524369 24646 hybrid_clock.cc:648] HybridClock initialized: now 1790492623524357 us; error 25 us; skew 500 ppm
Sep 27 07:03:43 dist-test-slave-mk3h krb5kdc[24633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:43.529832 24646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:43.530231 24646 webserver.cc:460] Webserver started at http://127.0.0.1:42617/ using document root <none> and password file <none>
I20260927 07:03:43.530443 24646 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:43.530508 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:43.530659 24646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:43.531700 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "61615ee1418b470995d11ab177c783c8"
format_stamp: "Formatted at 2026-09-27 07:03:43 on dist-test-slave-mk3h"
server_key: "78f771be03c828241f84e3b95d0f05ba"
server_key_iv: "aba0fa9dd9774920eddef5bb281038e8"
server_key_version: "encryptionkey@0"
I20260927 07:03:43.532060 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "61615ee1418b470995d11ab177c783c8"
format_stamp: "Formatted at 2026-09-27 07:03:43 on dist-test-slave-mk3h"
server_key: "78f771be03c828241f84e3b95d0f05ba"
server_key_iv: "aba0fa9dd9774920eddef5bb281038e8"
server_key_version: "encryptionkey@0"
I20260927 07:03:43.533325 24646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:43.534085 24661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:43.534262 24646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:43.534364 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "61615ee1418b470995d11ab177c783c8"
format_stamp: "Formatted at 2026-09-27 07:03:43 on dist-test-slave-mk3h"
server_key: "78f771be03c828241f84e3b95d0f05ba"
server_key_iv: "aba0fa9dd9774920eddef5bb281038e8"
server_key_version: "encryptionkey@0"
I20260927 07:03:43.534435 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:43.550112 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:43.551829 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:43.551980 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:43.555601 24646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36719
I20260927 07:03:43.555956 24713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36719 every 8 connection(s)
I20260927 07:03:43.555962 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:43.556587 24714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:43.558693 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8: Bootstrap starting.
I20260927 07:03:43.559240 24714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:43.559518 24714 log.cc:824] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:43.560269 24714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8: No bootstrap required, opened a new log
I20260927 07:03:43.561795 24714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260927 07:03:43.561920 24714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:43.561942 24714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61615ee1418b470995d11ab177c783c8, State: Initialized, Role: FOLLOWER
I20260927 07:03:43.562101 24714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260927 07:03:43.562184 24714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:43.562223 24714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:43.562273 24714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:43.562558 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24646
I20260927 07:03:43.562685 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:43.562872 3300 external_mini_cluster.cc:1428] Setting key 52dd5b9429e2020e35aec99377252f90
I20260927 07:03:43.562870 24714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260927 07:03:43.562965 24714 leader_election.cc:302] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61615ee1418b470995d11ab177c783c8; no voters:
I20260927 07:03:43.563133 24714 leader_election.cc:288] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:43.563156 24717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:43.563269 24717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [term 1 LEADER]: Becoming Leader. State: Replica: 61615ee1418b470995d11ab177c783c8, State: Running, Role: LEADER
I20260927 07:03:43.563377 24714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:43.563408 24717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } }
I20260927 07:03:43.563798 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61615ee1418b470995d11ab177c783c8. Latest consensus state: current_term: 1 leader_uuid: "61615ee1418b470995d11ab177c783c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } } }
I20260927 07:03:43.563901 24718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61615ee1418b470995d11ab177c783c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61615ee1418b470995d11ab177c783c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36719 } } }
I20260927 07:03:43.563944 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:43.564055 24718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:43.564352 24728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:43.564952 24728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:43.566927 24728 catalog_manager.cc:1347] Generated new cluster ID: f219103877df41bf8c2bd8decbd49a00
I20260927 07:03:43.566988 24728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:43.568946 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:43.563510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:46799 (local address 127.0.0.1:36719)
0927 07:03:43.563803 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:43.563824 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:43.563858 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:03:43.563978 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:43.563985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:43.564044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:43.564183 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:43.564391 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:43.565799 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:43.566804 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:43.567425 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:43.567538 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:43.568242 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:43.568245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:43.568246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:43.568443 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:43.568479 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:43.568495 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:43.568595 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:43.568702 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:03:43.568728 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 07:03:43.569854 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:43.575191 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:43.571406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:56799 (local address 127.0.0.1:36719)
0927 07:03:43.571445 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:43.571446 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:43.571681 (+ 235us) server_negotiation.cc:407] Connection header received
0927 07:03:43.571746 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:43.571747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:43.571757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:43.571813 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:43.571932 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:43.572938 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:43.573946 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:43.574249 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:43.574304 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:43.574936 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:43.574937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:43.574937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:43.574941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:43.574953 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:43.574957 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:43.575022 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:43.575116 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 07:03:43.575127 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:43.600131 24728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:43.600858 24728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:43.612828 24728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61615ee1418b470995d11ab177c783c8: Generated new TSK 0
I20260927 07:03:43.613034 24728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:43.615482 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24646
W20260927 07:03:43.620086 24631 connection.cc:535] client connection to 127.0.0.1:36719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36719): Connection reset by peer (error 104)
2026-09-27T07:03:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:43 dist-test-slave-mk3h krb5kdc[24745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:43 dist-test-slave-mk3h krb5kdc[24745](info): set up 2 sockets
Sep 27 07:03:43 dist-test-slave-mk3h krb5kdc[24745](info): commencing operation
krb5kdc: starting...
W20260927 07:03:44.647527 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:44 dist-test-slave-mk3h krb5kdc[24745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:03:44.754417 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:44.831102 24758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:44.831290 24758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:44.831311 24758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:44.832392 24758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:44.832432 24758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:44.832445 24758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:44.832484 24758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:44.832509 24758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:44.833935 24758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:44.834157 24758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:44.834343 24758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:44.838946 24764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:44.839005 24758 server_base.cc:1034] running on GCE node
W20260927 07:03:44.838959 24763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:44.838981 24766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:44.839360 24758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:44.839705 24758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:44.840884 24758 hybrid_clock.cc:648] HybridClock initialized: now 1790492624840872 us; error 44 us; skew 500 ppm
Sep 27 07:03:44 dist-test-slave-mk3h krb5kdc[24745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:44.843240 24758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:03:44.843720 24758 webserver.cc:460] Webserver started at http://127.0.0.1:33775/ using document root <none> and password file <none>
I20260927 07:03:44.843948 24758 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:44.844015 24758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:44.844153 24758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:44.845110 24758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "c6627170ad9248478962c688d5568dd5"
format_stamp: "Formatted at 2026-09-27 07:03:44 on dist-test-slave-mk3h"
server_key: "a646f9094859def74afc710353934364"
server_key_iv: "4848da7201a3e135ce274d33a28e246f"
server_key_version: "encryptionkey@0"
I20260927 07:03:44.845454 24758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "c6627170ad9248478962c688d5568dd5"
format_stamp: "Formatted at 2026-09-27 07:03:44 on dist-test-slave-mk3h"
server_key: "a646f9094859def74afc710353934364"
server_key_iv: "4848da7201a3e135ce274d33a28e246f"
server_key_version: "encryptionkey@0"
I20260927 07:03:44.847019 24758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:44.847774 24773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:44.847958 24758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 07:03:44.848067 24758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "c6627170ad9248478962c688d5568dd5"
format_stamp: "Formatted at 2026-09-27 07:03:44 on dist-test-slave-mk3h"
server_key: "a646f9094859def74afc710353934364"
server_key_iv: "4848da7201a3e135ce274d33a28e246f"
server_key_version: "encryptionkey@0"
I20260927 07:03:44.848157 24758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:44.851217 24758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:44.853440 24758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:44.853605 24758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:44.857424 24758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45275
I20260927 07:03:44.857452 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45275 every 8 connection(s)
I20260927 07:03:44.857749 24758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:44.858377 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24758
I20260927 07:03:44.858500 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:44.858681 3300 external_mini_cluster.cc:1428] Setting key 8c6cd3236273f4dd60d65b2979b9694e
I20260927 07:03:44.858867 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:44.861001 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:44.859180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:36301 (local address 127.0.0.1:45275)
0927 07:03:44.859359 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:44.859366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:44.859398 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:03:44.859943 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:44.859950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:44.859971 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:44.860073 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:44.860588 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:44.860591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:44.860591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:44.860595 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:44.860624 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:44.860633 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:44.860682 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:44.860765 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 07:03:44.860794 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260927 07:03:44.861872 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:44.862233 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5: Bootstrap starting.
I20260927 07:03:44.862908 24826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:44.863322 24826 log.cc:824] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:44.863972 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5: No bootstrap required, opened a new log
I20260927 07:03:44.864550 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:44.863416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.156:38095 (local address 127.0.0.1:45275)
0927 07:03:44.863447 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:44.863449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:44.863597 (+ 148us) server_negotiation.cc:407] Connection header received
0927 07:03:44.863720 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:44.863721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:44.863726 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:44.863782 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:44.864325 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:44.864326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:44.864326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:44.864331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:44.864342 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:44.864346 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:44.864399 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:44.864452 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:03:44.864465 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 07:03:44.865789 24826 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } }
I20260927 07:03:44.865932 24826 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:44.865978 24826 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6627170ad9248478962c688d5568dd5, State: Initialized, Role: FOLLOWER
I20260927 07:03:44.866097 24826 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } }
I20260927 07:03:44.866194 24826 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:44.866236 24826 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:44.866285 24826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:44.866768 24826 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } }
I20260927 07:03:44.866888 24826 leader_election.cc:302] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6627170ad9248478962c688d5568dd5; no voters:
I20260927 07:03:44.867046 24826 leader_election.cc:288] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:44.867079 24836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:44.867249 24836 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [term 1 LEADER]: Becoming Leader. State: Replica: c6627170ad9248478962c688d5568dd5, State: Running, Role: LEADER
I20260927 07:03:44.867268 24826 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:44.867363 24836 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } }
I20260927 07:03:44.867775 24837 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6627170ad9248478962c688d5568dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } } }
I20260927 07:03:44.867779 24838 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6627170ad9248478962c688d5568dd5. Latest consensus state: current_term: 1 leader_uuid: "c6627170ad9248478962c688d5568dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6627170ad9248478962c688d5568dd5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45275 } } }
I20260927 07:03:44.867933 24837 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:44.867944 24838 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:44.868362 24845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:44.868808 24845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:44.870172 24845 catalog_manager.cc:1347] Generated new cluster ID: f276d52fcdf647368e2deea0ab11bcbe
I20260927 07:03:44.870220 24845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:44.886173 24845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:44.886337 24845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:44.904836 24845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6627170ad9248478962c688d5568dd5: Generated new TSK 0
I20260927 07:03:44.905016 24845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:44.912819 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24758
2026-09-27T07:03:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:44 dist-test-slave-mk3h krb5kdc[24857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:44 dist-test-slave-mk3h krb5kdc[24857](info): set up 2 sockets
Sep 27 07:03:44 dist-test-slave-mk3h krb5kdc[24857](info): commencing operation
krb5kdc: starting...
W20260927 07:03:45.947714 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:46.060060 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42661
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:32879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:46.138159 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:46.138384 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:46.138418 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:46.139534 24870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:46.139618 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:46.139644 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:46.139678 24870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:46.139701 24870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:46.141152 24870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:32879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:42661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:46.141417 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:46.141649 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:46.146163 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:46.146201 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:46.146169 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:46.146224 24870 server_base.cc:1034] running on GCE node
I20260927 07:03:46.146564 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:46.146777 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:46.147936 24870 hybrid_clock.cc:648] HybridClock initialized: now 1790492626147930 us; error 35 us; skew 500 ppm
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:46.150202 24870 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:46.150583 24870 webserver.cc:460] Webserver started at http://127.3.57.62:40181/ using document root <none> and password file <none>
I20260927 07:03:46.150799 24870 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:46.150863 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:46.151027 24870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:46.152050 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "81000a49454b484399ffdfa7005276b7"
format_stamp: "Formatted at 2026-09-27 07:03:46 on dist-test-slave-mk3h"
server_key: "ac266a3dd502d1a2a841e99d37a43fcd"
server_key_iv: "a98e4f8e51c4fe8779c3234a32b68dd8"
server_key_version: "encryptionkey@0"
I20260927 07:03:46.152369 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "81000a49454b484399ffdfa7005276b7"
format_stamp: "Formatted at 2026-09-27 07:03:46 on dist-test-slave-mk3h"
server_key: "ac266a3dd502d1a2a841e99d37a43fcd"
server_key_iv: "a98e4f8e51c4fe8779c3234a32b68dd8"
server_key_version: "encryptionkey@0"
I20260927 07:03:46.153721 24870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:46.154413 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:46.154572 24870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:46.154665 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "81000a49454b484399ffdfa7005276b7"
format_stamp: "Formatted at 2026-09-27 07:03:46 on dist-test-slave-mk3h"
server_key: "ac266a3dd502d1a2a841e99d37a43fcd"
server_key_iv: "a98e4f8e51c4fe8779c3234a32b68dd8"
server_key_version: "encryptionkey@0"
I20260927 07:03:46.154742 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:46.163520 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:46.165174 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:46.165316 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:46.169045 24870 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:42661
I20260927 07:03:46.169076 24937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:42661 every 8 connection(s)
I20260927 07:03:46.169374 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:46.170225 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:46.172241 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7: Bootstrap starting.
I20260927 07:03:46.172729 24938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:46.172957 24938 log.cc:824] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:46.173552 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7: No bootstrap required, opened a new log
I20260927 07:03:46.174048 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24870
I20260927 07:03:46.174167 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:46.174330 3300 external_mini_cluster.cc:1428] Setting key 860c4017ff28fb88826bc3b71d8e15e7
I20260927 07:03:46.176041 24938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } }
I20260927 07:03:46.176256 24938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:46.176301 24938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81000a49454b484399ffdfa7005276b7, State: Initialized, Role: FOLLOWER
I20260927 07:03:46.176451 24938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } }
I20260927 07:03:46.176539 24938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:46.176579 24938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:46.176637 24938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:46.177263 24938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } }
I20260927 07:03:46.177385 24938 leader_election.cc:302] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81000a49454b484399ffdfa7005276b7; no voters:
I20260927 07:03:46.177585 24938 leader_election.cc:288] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:46.177605 24943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:46.177721 24943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [term 1 LEADER]: Becoming Leader. State: Replica: 81000a49454b484399ffdfa7005276b7, State: Running, Role: LEADER
I20260927 07:03:46.177843 24938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:46.177856 24943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } }
I20260927 07:03:46.178360 24944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81000a49454b484399ffdfa7005276b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } } }
I20260927 07:03:46.178623 24944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:46.178547 24945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81000a49454b484399ffdfa7005276b7. Latest consensus state: current_term: 1 leader_uuid: "81000a49454b484399ffdfa7005276b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81000a49454b484399ffdfa7005276b7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 42661 } } }
I20260927 07:03:46.179028 24952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:46.179040 24945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:46.179590 24952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:46.184656 24952 catalog_manager.cc:1347] Generated new cluster ID: 4702f764d0cc4ed4a135d295e2e75b4d
I20260927 07:03:46.184705 24952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:46.186180 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:46.174809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35369 (local address 127.3.57.62:42661)
0927 07:03:46.174966 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:46.174974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:46.174997 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:03:46.175964 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:46.175971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:46.176017 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:46.176124 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:46.177004 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:46.178153 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:46.179258 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:46.179867 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:46.179995 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:46.182520 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:46.182524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:46.182525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:46.182536 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:46.184426 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:46.184881 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:46.184884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:46.184885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:46.184987 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:46.185250 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:46.185250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:46.185251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:46.185614 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:46.185767 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:46.185872 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:03:46.185974 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260927 07:03:46.188539 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:46.196027 24952 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:46.196671 24952 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:46.197155 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:46.190015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52847 (local address 127.3.57.62:42661)
0927 07:03:46.190049 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:46.190051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:46.190059 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:46.190104 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:46.190104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:46.190113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:46.190175 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:46.190710 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:46.191455 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:46.192436 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:46.192776 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:46.192847 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:46.194099 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:46.194100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:46.194101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:46.194108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:46.195791 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:46.196200 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:46.196201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:46.196202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:46.196259 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:46.196492 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:46.196492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:46.196493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:46.196644 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:46.196743 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:46.196912 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 07:03:46.196975 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":18}
I20260927 07:03:46.200318 24952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81000a49454b484399ffdfa7005276b7: Generated new TSK 0
I20260927 07:03:46.200470 24952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:46.206228 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24870
W20260927 07:03:46.210250 24855 connection.cc:535] client connection to 127.3.57.62:42661 recv error: Network error: failed to read from TLS socket (remote: 127.3.57.62:42661): Connection reset by peer (error 104)
2026-09-27T07:03:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24969](info): set up 2 sockets
Sep 27 07:03:46 dist-test-slave-mk3h krb5kdc[24969](info): commencing operation
krb5kdc: starting...
W20260927 07:03:47.239681 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:47 dist-test-slave-mk3h krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:47.351871 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:33915
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:42561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:33915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:47.430137 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:47.430361 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:47.430399 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:47.431523 24982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:47.431645 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:47.431674 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:47.431702 24982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:47.431727 24982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:47.433204 24982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:42561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:33915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:33915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:47.433473 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:47.433696 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:47.438572 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:47.438573 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:47.438601 24988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:47.438917 24982 server_base.cc:1034] running on GCE node
I20260927 07:03:47.439138 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:47.439366 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:47.440528 24982 hybrid_clock.cc:648] HybridClock initialized: now 1790492627440495 us; error 44 us; skew 500 ppm
Sep 27 07:03:47 dist-test-slave-mk3h krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:47.442996 24982 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:47.443415 24982 webserver.cc:460] Webserver started at http://127.3.57.62:42471/ using document root <none> and password file <none>
I20260927 07:03:47.443655 24982 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:47.443701 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:47.443787 24982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:47.444718 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "0c41c086b6494d588646ed26ef1dd62b"
format_stamp: "Formatted at 2026-09-27 07:03:47 on dist-test-slave-mk3h"
server_key: "dd2ed7415f3d784c2702764ade6e06f8"
server_key_iv: "1a866ebd6ef0745e4839cb98c9ba1d9a"
server_key_version: "encryptionkey@0"
I20260927 07:03:47.445016 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "0c41c086b6494d588646ed26ef1dd62b"
format_stamp: "Formatted at 2026-09-27 07:03:47 on dist-test-slave-mk3h"
server_key: "dd2ed7415f3d784c2702764ade6e06f8"
server_key_iv: "1a866ebd6ef0745e4839cb98c9ba1d9a"
server_key_version: "encryptionkey@0"
I20260927 07:03:47.446228 24982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:47.446849 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:47.447011 24982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:47.447095 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "0c41c086b6494d588646ed26ef1dd62b"
format_stamp: "Formatted at 2026-09-27 07:03:47 on dist-test-slave-mk3h"
server_key: "dd2ed7415f3d784c2702764ade6e06f8"
server_key_iv: "1a866ebd6ef0745e4839cb98c9ba1d9a"
server_key_version: "encryptionkey@0"
I20260927 07:03:47.447162 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:47.474187 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:47.476182 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:47.476336 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:47.480324 24982 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:33915
I20260927 07:03:47.480358 25049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:33915 every 8 connection(s)
I20260927 07:03:47.480703 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:47.481536 25050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:47.483800 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: Bootstrap starting.
I20260927 07:03:47.484354 25050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:47.484638 25050 log.cc:824] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:47.485324 25050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: No bootstrap required, opened a new log
I20260927 07:03:47.486163 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 24982
I20260927 07:03:47.486248 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:47.486455 3300 external_mini_cluster.cc:1428] Setting key f704fd6b751752660d285c60f4442cd2
I20260927 07:03:47.487370 25050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } }
I20260927 07:03:47.487572 25050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:47.487614 25050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c41c086b6494d588646ed26ef1dd62b, State: Initialized, Role: FOLLOWER
I20260927 07:03:47.487780 25050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } }
I20260927 07:03:47.487900 25050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:47.487934 25050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:47.487985 25050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:47.488683 25050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } }
I20260927 07:03:47.488788 25050 leader_election.cc:302] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c41c086b6494d588646ed26ef1dd62b; no voters:
I20260927 07:03:47.488957 25050 leader_election.cc:288] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:47.489012 25055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:47.489184 25055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [term 1 LEADER]: Becoming Leader. State: Replica: 0c41c086b6494d588646ed26ef1dd62b, State: Running, Role: LEADER
I20260927 07:03:47.489187 25050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:47.489320 25055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } }
W20260927 07:03:47.491053 25069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:47.491151 25069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:47.491236 25056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c41c086b6494d588646ed26ef1dd62b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } } }
I20260927 07:03:47.491243 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c41c086b6494d588646ed26ef1dd62b. Latest consensus state: current_term: 1 leader_uuid: "0c41c086b6494d588646ed26ef1dd62b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c41c086b6494d588646ed26ef1dd62b" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 33915 } } }
I20260927 07:03:47.491380 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:47.491606 25056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:47.492812 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:47.486969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35487 (local address 127.3.57.62:33915)
0927 07:03:47.487192 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:47.487200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:47.487234 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:03:47.487343 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:47.487349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:47.487414 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:47.487591 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:47.487876 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:47.489547 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:47.490744 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:47.491389 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:47.491505 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:47.492177 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:47.492180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:47.492180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:47.492334 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:47.492362 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:47.492371 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:47.492438 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:47.492556 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 07:03:47.492604 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260927 07:03:47.493700 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:47.495682 25072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:47.495901 25072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:47.497462 25072 catalog_manager.cc:1347] Generated new cluster ID: c186c63505ec463fa3ad740d52037181
I20260927 07:03:47.497516 25072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:47.498272 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:47.495080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33915 (local address 127.3.57.62:33915)
0927 07:03:47.495116 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:47.495117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:47.495197 (+ 80us) server_negotiation.cc:407] Connection header received
0927 07:03:47.495255 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:47.495255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:47.495262 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:47.495301 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:47.495426 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:47.496054 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:47.496881 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:47.497177 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:47.497261 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:47.498027 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:47.498028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:47.498029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:47.498033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:47.498048 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:47.498052 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:47.498118 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:47.498191 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 07:03:47.498202 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:47.510579 25072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:47.511092 25072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:47.516875 25072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c41c086b6494d588646ed26ef1dd62b: Generated new TSK 0
I20260927 07:03:47.517092 25072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:47.526037 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 24982
W20260927 07:03:47.530270 24967 connection.cc:535] client connection to 127.3.57.62:33915 recv error: Network error: failed to read from TLS socket (remote: 127.3.57.62:33915): Connection reset by peer (error 104)
2026-09-27T07:03:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:47 dist-test-slave-mk3h krb5kdc[25081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:47 dist-test-slave-mk3h krb5kdc[25081](info): set up 2 sockets
Sep 27 07:03:47 dist-test-slave-mk3h krb5kdc[25081](info): commencing operation
krb5kdc: starting...
W20260927 07:03:48.551465 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:48 dist-test-slave-mk3h krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:48.662092 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36307
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:48.740864 25094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:48.741082 25094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:48.741119 25094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:48.742223 25094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:48.742286 25094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:48.742314 25094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:48.742347 25094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:48.742370 25094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:48.743777 25094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:36307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:48.744042 25094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:48.744259 25094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:48.749073 25100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:48.749099 25099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:48.749171 25094 server_base.cc:1034] running on GCE node
W20260927 07:03:48.749099 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:48.749570 25094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:48.749862 25094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:48.751012 25094 hybrid_clock.cc:648] HybridClock initialized: now 1790492628751005 us; error 38 us; skew 500 ppm
Sep 27 07:03:48 dist-test-slave-mk3h krb5kdc[25081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:48.758266 25094 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:48.758736 25094 webserver.cc:460] Webserver started at http://127.3.57.62:37287/ using document root <none> and password file <none>
I20260927 07:03:48.758980 25094 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:48.759053 25094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:48.759197 25094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:48.760317 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "1206d31f622d4895b19296d1141c7dab"
format_stamp: "Formatted at 2026-09-27 07:03:48 on dist-test-slave-mk3h"
server_key: "3bfa6c0915e8f560f14c705a4967956f"
server_key_iv: "38a623e0acf4ba57f320c4d83fc1584d"
server_key_version: "encryptionkey@0"
I20260927 07:03:48.760667 25094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "1206d31f622d4895b19296d1141c7dab"
format_stamp: "Formatted at 2026-09-27 07:03:48 on dist-test-slave-mk3h"
server_key: "3bfa6c0915e8f560f14c705a4967956f"
server_key_iv: "38a623e0acf4ba57f320c4d83fc1584d"
server_key_version: "encryptionkey@0"
I20260927 07:03:48.762033 25094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:03:48.762967 25109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:48.763298 25094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:48.763417 25094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "1206d31f622d4895b19296d1141c7dab"
format_stamp: "Formatted at 2026-09-27 07:03:48 on dist-test-slave-mk3h"
server_key: "3bfa6c0915e8f560f14c705a4967956f"
server_key_iv: "38a623e0acf4ba57f320c4d83fc1584d"
server_key_version: "encryptionkey@0"
I20260927 07:03:48.763497 25094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:48.766896 25094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:48.769717 25094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:48.769971 25094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:48.774948 25094 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:36307
I20260927 07:03:48.775089 25161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:36307 every 8 connection(s)
I20260927 07:03:48.775385 25094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:48.776077 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25094
I20260927 07:03:48.776161 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:48.776360 3300 external_mini_cluster.cc:1428] Setting key 11d046233fc2df4adb665a70634dbf45
I20260927 07:03:48.776715 25162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:48.779379 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:48.776908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58279 (local address 127.3.57.62:36307)
0927 07:03:48.777110 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:48.777119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:48.777152 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:03:48.777848 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:48.777858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:48.777893 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:48.778047 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:48.778904 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:48.778908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:48.778909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:48.778914 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:48.778959 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:48.778979 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:48.779055 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:48.779121 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 07:03:48.779163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260927 07:03:48.780992 25162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab: Bootstrap starting.
I20260927 07:03:48.781174 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:48.781894 25162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:48.782209 25162 log.cc:824] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:48.783136 25162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab: No bootstrap required, opened a new log
I20260927 07:03:48.784521 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:48.782677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34259 (local address 127.3.57.62:36307)
0927 07:03:48.782729 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:48.782730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:48.782834 (+ 104us) server_negotiation.cc:407] Connection header received
0927 07:03:48.782878 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:48.782878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:48.782883 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:48.782938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:48.784209 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:48.784210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:48.784210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:48.784214 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:48.784228 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:48.784232 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:48.784289 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:48.784369 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 07:03:48.784384 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:03:48.785501 25162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } }
I20260927 07:03:48.785681 25162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:48.785729 25162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1206d31f622d4895b19296d1141c7dab, State: Initialized, Role: FOLLOWER
I20260927 07:03:48.785856 25162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } }
I20260927 07:03:48.785966 25162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:48.786010 25162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:48.786060 25162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:48.786653 25162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } }
I20260927 07:03:48.786777 25162 leader_election.cc:302] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1206d31f622d4895b19296d1141c7dab; no voters:
I20260927 07:03:48.786979 25162 leader_election.cc:288] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:48.787098 25172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:48.787254 25162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:48.787310 25172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [term 1 LEADER]: Becoming Leader. State: Replica: 1206d31f622d4895b19296d1141c7dab, State: Running, Role: LEADER
I20260927 07:03:48.787438 25172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } }
I20260927 07:03:48.787873 25172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1206d31f622d4895b19296d1141c7dab. Latest consensus state: current_term: 1 leader_uuid: "1206d31f622d4895b19296d1141c7dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } } }
I20260927 07:03:48.787899 25173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1206d31f622d4895b19296d1141c7dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1206d31f622d4895b19296d1141c7dab" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 36307 } } }
I20260927 07:03:48.787974 25172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:48.787978 25173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:48.788686 25181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:48.789136 25181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:48.790724 25181 catalog_manager.cc:1347] Generated new cluster ID: 1ac2bfefb56341fbb7ff5e75f6f11dac
I20260927 07:03:48.790776 25181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:48.800609 25181 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:48.800812 25181 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:48.811307 25181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1206d31f622d4895b19296d1141c7dab: Generated new TSK 0
I20260927 07:03:48.811519 25181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:48.819430 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25094
2026-09-27T07:03:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:48 dist-test-slave-mk3h krb5kdc[25193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:48 dist-test-slave-mk3h krb5kdc[25193](info): set up 2 sockets
Sep 27 07:03:48 dist-test-slave-mk3h krb5kdc[25193](info): commencing operation
krb5kdc: starting...
W20260927 07:03:49.863780 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:49 dist-test-slave-mk3h krb5kdc[25193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:49.977890 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41473
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:50.055181 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:50.055402 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:50.055441 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:50.056531 25206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:50.056602 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:50.056629 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:50.056658 25206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:50.056676 25206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:50.058024 25206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:50.058275 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:50.058498 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:50.063045 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:50.063073 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:50.063174 25206 server_base.cc:1034] running on GCE node
W20260927 07:03:50.063380 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:50.063678 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:50.063896 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:50.065032 25206 hybrid_clock.cc:648] HybridClock initialized: now 1790492630065020 us; error 30 us; skew 500 ppm
Sep 27 07:03:50 dist-test-slave-mk3h krb5kdc[25193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:50.067371 25206 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:50.067862 25206 webserver.cc:460] Webserver started at http://127.3.57.62:41665/ using document root <none> and password file <none>
I20260927 07:03:50.068104 25206 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:50.068177 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:50.068329 25206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:50.069303 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "5607706b218746fd878a1c9bd2e0c570"
format_stamp: "Formatted at 2026-09-27 07:03:50 on dist-test-slave-mk3h"
server_key: "5d37493732a7663db3056c241fc41e91"
server_key_iv: "edc1afbe2e7730cf76ee156586f880a8"
server_key_version: "encryptionkey@0"
I20260927 07:03:50.069649 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "5607706b218746fd878a1c9bd2e0c570"
format_stamp: "Formatted at 2026-09-27 07:03:50 on dist-test-slave-mk3h"
server_key: "5d37493732a7663db3056c241fc41e91"
server_key_iv: "edc1afbe2e7730cf76ee156586f880a8"
server_key_version: "encryptionkey@0"
I20260927 07:03:50.070900 25206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:03:50.071630 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:50.071801 25206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:50.071903 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "5607706b218746fd878a1c9bd2e0c570"
format_stamp: "Formatted at 2026-09-27 07:03:50 on dist-test-slave-mk3h"
server_key: "5d37493732a7663db3056c241fc41e91"
server_key_iv: "edc1afbe2e7730cf76ee156586f880a8"
server_key_version: "encryptionkey@0"
I20260927 07:03:50.071992 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:50.100935 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:50.102694 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:50.102861 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:50.106755 25206 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41473
I20260927 07:03:50.106787 25273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41473 every 8 connection(s)
I20260927 07:03:50.107100 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:50.107859 25274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:50.109869 25274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: Bootstrap starting.
I20260927 07:03:50.110357 25274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:50.110579 25274 log.cc:824] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:50.111168 25274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: No bootstrap required, opened a new log
I20260927 07:03:50.112061 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25206
I20260927 07:03:50.112169 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:50.112349 3300 external_mini_cluster.cc:1428] Setting key 771d631d188d4c17992f460e35ee34bb
I20260927 07:03:50.113250 25274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } }
I20260927 07:03:50.113420 25274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:50.113463 25274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5607706b218746fd878a1c9bd2e0c570, State: Initialized, Role: FOLLOWER
I20260927 07:03:50.113593 25274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } }
I20260927 07:03:50.113679 25274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:50.113711 25274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:50.113762 25274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:50.114681 25274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } }
I20260927 07:03:50.114765 25274 leader_election.cc:302] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5607706b218746fd878a1c9bd2e0c570; no voters:
I20260927 07:03:50.114916 25274 leader_election.cc:288] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:50.114967 25279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:50.115127 25279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [term 1 LEADER]: Becoming Leader. State: Replica: 5607706b218746fd878a1c9bd2e0c570, State: Running, Role: LEADER
I20260927 07:03:50.115187 25274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:50.115249 25279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } }
I20260927 07:03:50.115739 25280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5607706b218746fd878a1c9bd2e0c570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } } }
I20260927 07:03:50.115830 25280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:50.116158 25281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5607706b218746fd878a1c9bd2e0c570. Latest consensus state: current_term: 1 leader_uuid: "5607706b218746fd878a1c9bd2e0c570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5607706b218746fd878a1c9bd2e0c570" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41473 } } }
I20260927 07:03:50.116262 25281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570 [sys.catalog]: This master's current role is: LEADER
W20260927 07:03:50.116844 25295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:50.116896 25295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:50.116955 25288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:50.117134 25288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:50.118832 25288 catalog_manager.cc:1347] Generated new cluster ID: 06e165bd2b7d4e96b763ef5380c0413a
I20260927 07:03:50.118876 25288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 07:03:50 dist-test-slave-mk3h krb5kdc[25193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:50.126902 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:50.112860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58125 (local address 127.3.57.62:41473)
0927 07:03:50.113032 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:50.113039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:50.113064 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:03:50.113212 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:50.113217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:50.113255 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:50.113354 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:50.114184 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:50.115225 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:50.116362 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:50.116939 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:50.117330 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:50.119735 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:50.119739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:50.119739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:50.119749 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:50.125282 (+ 5533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:50.125693 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:50.125696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:50.125696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:50.125807 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:50.126044 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:50.126044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:50.126045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:50.126301 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:50.126439 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:50.126595 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 07:03:50.126715 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260927 07:03:50.127830 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:50.133141 25288 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:50.133689 25288 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:50.138665 25288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5607706b218746fd878a1c9bd2e0c570: Generated new TSK 0
I20260927 07:03:50.138823 25288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:50.139447 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:50.131834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57121 (local address 127.3.57.62:41473)
0927 07:03:50.131866 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:50.131868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:50.131978 (+ 110us) server_negotiation.cc:407] Connection header received
0927 07:03:50.132031 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:50.132032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:50.132041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:50.132098 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:50.132667 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:50.133408 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:50.134363 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:50.134697 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:50.134774 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:50.136002 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:50.136003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:50.136004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:50.136011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:50.138188 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:50.138682 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:50.138683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:50.138683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:50.138741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:50.138936 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:50.138937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:50.138938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:50.139060 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:50.139189 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:50.139271 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 07:03:50.139353 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:03:50.141331 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25206
2026-09-27T07:03:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:50 dist-test-slave-mk3h krb5kdc[25305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:50 dist-test-slave-mk3h krb5kdc[25305](info): set up 2 sockets
Sep 27 07:03:50 dist-test-slave-mk3h krb5kdc[25305](info): commencing operation
krb5kdc: starting...
W20260927 07:03:51.171890 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:51 dist-test-slave-mk3h krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:51.297602 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:46069
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:51.377918 25318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:51.378191 25318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:51.378230 25318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:51.379405 25318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:51.379491 25318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:51.379516 25318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:51.379544 25318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:51.379601 25318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:51.381147 25318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:51.381436 25318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:51.381680 25318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:51.386540 25326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:51.386679 25324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:51.386660 25318 server_base.cc:1034] running on GCE node
W20260927 07:03:51.386778 25323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:51.387099 25318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:51.387427 25318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:51.388581 25318 hybrid_clock.cc:648] HybridClock initialized: now 1790492631388580 us; error 59 us; skew 500 ppm
Sep 27 07:03:51 dist-test-slave-mk3h krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:51.391885 25318 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:51.392870 25318 webserver.cc:460] Webserver started at http://127.3.57.62:39273/ using document root <none> and password file <none>
I20260927 07:03:51.393193 25318 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:51.393285 25318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:51.393450 25318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:51.394906 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "81791d76da024254a37af65f67767ca2"
format_stamp: "Formatted at 2026-09-27 07:03:51 on dist-test-slave-mk3h"
server_key: "6d383e4fd462c513c933f382d8aab461"
server_key_iv: "af9622317ddf834860d23da0a9bf7fe0"
server_key_version: "encryptionkey@0"
I20260927 07:03:51.395422 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "81791d76da024254a37af65f67767ca2"
format_stamp: "Formatted at 2026-09-27 07:03:51 on dist-test-slave-mk3h"
server_key: "6d383e4fd462c513c933f382d8aab461"
server_key_iv: "af9622317ddf834860d23da0a9bf7fe0"
server_key_version: "encryptionkey@0"
I20260927 07:03:51.398838 25318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:03:51.400156 25333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:51.400353 25318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:51.400456 25318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "81791d76da024254a37af65f67767ca2"
format_stamp: "Formatted at 2026-09-27 07:03:51 on dist-test-slave-mk3h"
server_key: "6d383e4fd462c513c933f382d8aab461"
server_key_iv: "af9622317ddf834860d23da0a9bf7fe0"
server_key_version: "encryptionkey@0"
I20260927 07:03:51.400523 25318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:51.417785 25318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:51.420812 25318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:51.421022 25318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:51.426644 25318 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:46069
I20260927 07:03:51.426707 25385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:46069 every 8 connection(s)
I20260927 07:03:51.427098 25318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:51.428251 25386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:51.430975 25386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: Bootstrap starting.
I20260927 07:03:51.431705 25386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:51.432174 25386 log.cc:824] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:51.432405 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25318
I20260927 07:03:51.432550 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:51.432767 3300 external_mini_cluster.cc:1428] Setting key 47121465fe48ef39e319d9a8f2809e4b
I20260927 07:03:51.433022 25386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: No bootstrap required, opened a new log
I20260927 07:03:51.436090 25386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } }
I20260927 07:03:51.436275 25386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:51.436312 25386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81791d76da024254a37af65f67767ca2, State: Initialized, Role: FOLLOWER
I20260927 07:03:51.436432 25386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } }
I20260927 07:03:51.436511 25386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:51.436540 25386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:51.436595 25386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:51.437263 25386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } }
I20260927 07:03:51.437351 25386 leader_election.cc:302] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81791d76da024254a37af65f67767ca2; no voters:
I20260927 07:03:51.437597 25386 leader_election.cc:288] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:51.437618 25391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:51.437731 25391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [term 1 LEADER]: Becoming Leader. State: Replica: 81791d76da024254a37af65f67767ca2, State: Running, Role: LEADER
I20260927 07:03:51.437866 25386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:51.437878 25391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } }
I20260927 07:03:51.438241 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81791d76da024254a37af65f67767ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } } }
I20260927 07:03:51.438388 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:51.438552 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81791d76da024254a37af65f67767ca2. Latest consensus state: current_term: 1 leader_uuid: "81791d76da024254a37af65f67767ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81791d76da024254a37af65f67767ca2" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 46069 } } }
I20260927 07:03:51.438619 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2 [sys.catalog]: This master's current role is: LEADER
W20260927 07:03:51.439826 25407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:51.439881 25407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:51.439936 25399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:51.440124 25399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:51.440551 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:51.433225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43089 (local address 127.3.57.62:46069)
0927 07:03:51.433396 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:51.433412 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:51.433548 (+ 136us) server_negotiation.cc:407] Connection header received
0927 07:03:51.434546 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:51.434553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:51.434602 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:51.434711 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:51.435032 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:51.435976 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:51.437105 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:51.437618 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:51.437779 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:51.439417 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:51.439421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:51.439421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:51.440063 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:51.440100 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:51.440121 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:51.440202 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:51.440278 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 07:03:51.440342 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260927 07:03:51.441509 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:51.441684 25399 catalog_manager.cc:1347] Generated new cluster ID: 2168f2726a0c4117bed4ab0a09899c90
I20260927 07:03:51.441723 25399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:51.446875 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:51.443634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59355 (local address 127.3.57.62:46069)
0927 07:03:51.443673 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:51.443675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:51.443684 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:51.443703 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:51.443704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:51.443714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:51.443776 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:51.443924 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:51.444650 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:51.445623 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:51.445925 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:51.445999 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:51.446624 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:51.446625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:51.446625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:51.446630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:51.446646 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:51.446650 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:51.446709 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:51.446778 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 07:03:51.446800 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:03:51.452376 25399 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:51.453071 25399 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:51.469902 25399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81791d76da024254a37af65f67767ca2: Generated new TSK 0
I20260927 07:03:51.470113 25399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:51.478124 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25318
2026-09-27T07:03:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:51 dist-test-slave-mk3h krb5kdc[25417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:51 dist-test-slave-mk3h krb5kdc[25417](info): set up 2 sockets
Sep 27 07:03:51 dist-test-slave-mk3h krb5kdc[25417](info): commencing operation
krb5kdc: starting...
W20260927 07:03:52.515780 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:52 dist-test-slave-mk3h krb5kdc[25417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:52.632468 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41059
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:41059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:52.713685 25430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:52.713872 25430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:52.713896 25430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:52.714938 25430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:52.714972 25430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:52.714986 25430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:52.714998 25430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:52.715011 25430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:52.716464 25430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:41059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:52.716645 25430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:52.716799 25430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:52.721504 25435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:52.721509 25430 server_base.cc:1034] running on GCE node
W20260927 07:03:52.721702 25436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:52.721721 25438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:52.722021 25430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:52.722316 25430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:52.723503 25430 hybrid_clock.cc:648] HybridClock initialized: now 1790492632723507 us; error 48 us; skew 500 ppm
Sep 27 07:03:52 dist-test-slave-mk3h krb5kdc[25417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:52.726001 25430 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:52.726469 25430 webserver.cc:460] Webserver started at http://127.3.57.62:45121/ using document root <none> and password file <none>
I20260927 07:03:52.726696 25430 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:52.726768 25430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:52.726930 25430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:52.728010 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "73736099d84444e6b49c419f14b71995"
format_stamp: "Formatted at 2026-09-27 07:03:52 on dist-test-slave-mk3h"
server_key: "e5bf80dac00a09428965edc0432e4ebd"
server_key_iv: "8753a7c8ac7cda2defd6448f29edfced"
server_key_version: "encryptionkey@0"
I20260927 07:03:52.728377 25430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "73736099d84444e6b49c419f14b71995"
format_stamp: "Formatted at 2026-09-27 07:03:52 on dist-test-slave-mk3h"
server_key: "e5bf80dac00a09428965edc0432e4ebd"
server_key_iv: "8753a7c8ac7cda2defd6448f29edfced"
server_key_version: "encryptionkey@0"
I20260927 07:03:52.729678 25430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:52.730379 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:52.730551 25430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:52.730655 25430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "73736099d84444e6b49c419f14b71995"
format_stamp: "Formatted at 2026-09-27 07:03:52 on dist-test-slave-mk3h"
server_key: "e5bf80dac00a09428965edc0432e4ebd"
server_key_iv: "8753a7c8ac7cda2defd6448f29edfced"
server_key_version: "encryptionkey@0"
I20260927 07:03:52.730739 25430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:52.733844 25430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:52.735805 25430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:52.735908 25430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:52.739485 25430 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:41059
I20260927 07:03:52.739573 25497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:41059 every 8 connection(s)
I20260927 07:03:52.739863 25430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:52.740801 25498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:52.742898 25498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995: Bootstrap starting.
I20260927 07:03:52.743451 25498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:52.743810 25498 log.cc:824] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:52.744498 25498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995: No bootstrap required, opened a new log
I20260927 07:03:52.746256 25498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } }
I20260927 07:03:52.746415 25498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:03:52.746462 25498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73736099d84444e6b49c419f14b71995, State: Initialized, Role: FOLLOWER
I20260927 07:03:52.746647 25498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } }
I20260927 07:03:52.746752 25498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:03:52.746796 25498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:03:52.746843 25498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:52.747054 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25430
I20260927 07:03:52.747177 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:52.747382 3300 external_mini_cluster.cc:1428] Setting key cf95aaf0ea202368a34fc7ea69046497
I20260927 07:03:52.747941 25498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } }
I20260927 07:03:52.748057 25498 leader_election.cc:302] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73736099d84444e6b49c419f14b71995; no voters:
I20260927 07:03:52.748224 25498 leader_election.cc:288] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:03:52.748291 25503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:52.748462 25503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [term 1 LEADER]: Becoming Leader. State: Replica: 73736099d84444e6b49c419f14b71995, State: Running, Role: LEADER
I20260927 07:03:52.748574 25498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:52.748590 25503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } }
I20260927 07:03:52.748998 25503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73736099d84444e6b49c419f14b71995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } } }
I20260927 07:03:52.749105 25503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:52.749279 25503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73736099d84444e6b49c419f14b71995. Latest consensus state: current_term: 1 leader_uuid: "73736099d84444e6b49c419f14b71995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73736099d84444e6b49c419f14b71995" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 41059 } } }
I20260927 07:03:52.749405 25503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:52.750007 25509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:52.750072 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:52.747871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33523 (local address 127.3.57.62:41059)
0927 07:03:52.748159 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:52.748168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:52.748199 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:03:52.748307 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:52.748313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:52.748344 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:52.749217 (+ 873us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:52.749509 (+ 292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:52.749514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:52.749514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:52.749518 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:52.749554 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:52.749567 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:52.749636 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:52.749840 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 07:03:52.749881 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":70,"threads_started":1}
I20260927 07:03:52.750579 25509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:52.751353 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:52.752306 25509 catalog_manager.cc:1347] Generated new cluster ID: bb2fb6a06be74c96b8bcd176d164ce07
I20260927 07:03:52.752354 25509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:52.756366 25509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:52.756503 25509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:52.759980 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:52.755654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46315 (local address 127.3.57.62:41059)
0927 07:03:52.755715 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:52.755717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:52.755728 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:03:52.755749 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:52.755750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:52.755755 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:52.755825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:52.759723 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:52.759725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:52.759725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:03:52.759730 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:52.759748 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:03:52.759753 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:52.759822 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:52.759899 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 07:03:52.759915 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 07:03:52.760041 25509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73736099d84444e6b49c419f14b71995: Generated new TSK 0
I20260927 07:03:52.760185 25509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:03:52.762351 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25430
2026-09-27T07:03:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1284 ms)
[----------] 18 tests from AuthTokenIssuingTest (23434 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:52 dist-test-slave-mk3h krb5kdc[25529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:52 dist-test-slave-mk3h krb5kdc[25529](info): set up 2 sockets
Sep 27 07:03:52 dist-test-slave-mk3h krb5kdc[25529](info): commencing operation
krb5kdc: starting...
W20260927 07:03:53.799787 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:53 dist-test-slave-mk3h krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:53.917943 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44195
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:53.997638 25542 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:53.997862 25542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:53.997879 25542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:53.997898 25542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:53.999131 25542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:53.999176 25542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:53.999189 25542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:53.999202 25542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:53.999214 25542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:54.000705 25542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:44195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:54.000949 25542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:54.001170 25542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:54.006174 25548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.006260 25542 server_base.cc:1034] running on GCE node
W20260927 07:03:54.006170 25550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:54.006194 25547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.006675 25542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:54.006978 25542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:54.008157 25542 hybrid_clock.cc:648] HybridClock initialized: now 1790492634008153 us; error 39 us; skew 500 ppm
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:54.014011 25542 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:54.014482 25542 webserver.cc:460] Webserver started at http://127.3.57.62:41293/ using document root <none> and password file <none>
I20260927 07:03:54.014703 25542 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:54.014772 25542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:54.014925 25542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:54.016018 25542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "edeee2b3e4ae4f43af44c59c3d40c286"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "e1074c7003a0ecc614f960e396b08aa4"
server_key_iv: "3696e9640f2f5211aaed9b8d687990aa"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.016366 25542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "edeee2b3e4ae4f43af44c59c3d40c286"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "e1074c7003a0ecc614f960e396b08aa4"
server_key_iv: "3696e9640f2f5211aaed9b8d687990aa"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.017659 25542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:54.018383 25557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:54.018571 25542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:54.018672 25542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "edeee2b3e4ae4f43af44c59c3d40c286"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "e1074c7003a0ecc614f960e396b08aa4"
server_key_iv: "3696e9640f2f5211aaed9b8d687990aa"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.018754 25542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:54.028506 25542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:54.030644 25542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:54.030803 25542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:54.034782 25542 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:44195
I20260927 07:03:54.034832 25609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:44195 every 8 connection(s)
I20260927 07:03:54.035192 25542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:54.036185 25610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:54.037616 25610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:54.042508 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25542
I20260927 07:03:54.042627 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:54.042819 3300 external_mini_cluster.cc:1428] Setting key cb2d665a298ac6ec3ed34ac9bc9aa08e
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:54.047791 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.038173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44195 (local address 127.0.0.1:51972)
0927 07:03:54.038397 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.038439 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.038930 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.039223 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.039226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.039498 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.039657 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.039662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.040689 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.040690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.041348 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.041351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.041865 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.041867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.041928 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.042451 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.042458 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.044113 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.046173 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.046178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.046180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.046362 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.046653 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.046654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.046654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.046738 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.047304 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.047310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.047371 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.047456 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.047646 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
I20260927 07:03:54.047950 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.038253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.3.57.62:44195)
0927 07:03:54.038406 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.038415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.038550 (+ 135us) server_negotiation.cc:407] Connection header received
0927 07:03:54.039100 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.039102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.039118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.039216 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.039766 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.040574 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.041453 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.041744 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.041881 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.044219 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.044226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.044227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.044235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.046089 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.046453 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.046457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.046457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.046563 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.046820 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.046821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.046821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.047102 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.047194 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.047527 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.047762 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260927 07:03:54.048748 25610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:54.049091 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.048979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48216)
0927 07:03:54.049030 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.049073 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 07:03:54.049259 25559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
I20260927 07:03:54.049674 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.049588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48224)
0927 07:03:54.049635 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.049658 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 07:03:54.049818 25610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:43993: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.57.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
I20260927 07:03:54.066821 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:43993
--webserver_interface=127.3.57.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:54.099095 25610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } attempt: 1
I20260927 07:03:54.099663 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.099482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48240)
0927 07:03:54.099583 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.099629 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 07:03:54.100069 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.099975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48254)
0927 07:03:54.100020 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.100046 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:03:54.100193 25610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:43993: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
W20260927 07:03:54.150357 25617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:54.150588 25617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:54.150616 25617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:54.150658 25617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:54.151844 25617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:54.151906 25617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:54.151929 25617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:54.151960 25617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:54.151981 25617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:54.153337 25617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.57.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:54.153585 25617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:54.153795 25617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:54.158327 25622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.158378 25617 server_base.cc:1034] running on GCE node
W20260927 07:03:54.158320 25623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:54.158317 25625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.158764 25617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:54.159049 25617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:54.160213 25617 hybrid_clock.cc:648] HybridClock initialized: now 1790492634160190 us; error 34 us; skew 500 ppm
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:54.162694 25617 init.cc:377] Logged in from keytab as kudu/127.3.57.61@KRBTEST.COM (short username kudu)
I20260927 07:03:54.163105 25617 webserver.cc:460] Webserver started at http://127.3.57.61:41633/ using document root <none> and password file <none>
I20260927 07:03:54.163323 25617 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:54.163395 25617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:54.163589 25617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:54.164543 25617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data/instance:
uuid: "861a7b0bc2be4835b146db9842581351"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "a539cd79e43dbf27592160bd5810e844"
server_key_iv: "95e8c3a0131dbdc0458d25c8f2a3a9fe"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.164903 25617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal/instance:
uuid: "861a7b0bc2be4835b146db9842581351"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "a539cd79e43dbf27592160bd5810e844"
server_key_iv: "95e8c3a0131dbdc0458d25c8f2a3a9fe"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.166164 25617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:03:54.166847 25632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:54.167023 25617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:54.167119 25617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "861a7b0bc2be4835b146db9842581351"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "a539cd79e43dbf27592160bd5810e844"
server_key_iv: "95e8c3a0131dbdc0458d25c8f2a3a9fe"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.167199 25617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:54.168339 25610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } attempt: 2
I20260927 07:03:54.168776 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.168591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48268)
0927 07:03:54.168659 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.168714 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 07:03:54.169180 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.169060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48272)
0927 07:03:54.169114 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.169146 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 07:03:54.169283 25610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:43993: Network error: Client connection negotiation failed: client connection to 127.3.57.61:43993: connect: Connection refused (error 111)
I20260927 07:03:54.204010 25617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:54.206401 25617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:54.206570 25617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:54.210624 25617 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.61:43993
I20260927 07:03:54.210688 25684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.61:43993 every 8 connection(s)
I20260927 07:03:54.211035 25617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
I20260927 07:03:54.211628 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25617
I20260927 07:03:54.211763 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-1/wal/instance
I20260927 07:03:54.211953 3300 external_mini_cluster.cc:1428] Setting key 8f13e753ce17950d730b4a97723ac26e
I20260927 07:03:54.212150 25685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:54.213670 25685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.57.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.60@KRBTEST.COM" created.
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:54.227110 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.3.57.62:44195)
0927 07:03:54.214486 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.214488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.214709 (+ 221us) server_negotiation.cc:407] Connection header received
0927 07:03:54.215462 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.215463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.215479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.215568 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.216393 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.216896 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.218451 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.218649 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.218700 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.224015 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.224017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.224017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.224026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.225629 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.226103 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.226105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.226105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.226167 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.226441 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.226442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.226442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.226564 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.226646 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.226927 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.227014 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:03:54.227308 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.214365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44195 (local address 127.0.0.1:51974)
0927 07:03:54.214569 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.214614 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.215303 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.215667 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.215672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.216014 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.216288 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.216295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.217008 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.217009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.218358 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.218361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.218751 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.218753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.218916 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.219393 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.219403 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.223894 (+ 4491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.225743 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.225751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.225753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.225980 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.226241 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.226242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.226242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.226357 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.226637 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.226644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.226790 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.226906 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.227064 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260927 07:03:54.228055 25685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:54.236340 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.228351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48274)
0927 07:03:54.228573 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.228585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.228641 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.228816 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.228817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.228984 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.229049 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.229051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.229870 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.229871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.230723 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.230726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.231179 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.231180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.231234 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.231650 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.231654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.233448 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.235305 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.235306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.235307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.235473 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.235782 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.235783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.235783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.235847 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.236105 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.236105 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.236137 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.236182 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.236281 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 07:03:54.236512 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.228431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.3.57.61:43993)
0927 07:03:54.228588 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.228592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.228633 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:03:54.228719 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.228721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.228731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.228808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.229122 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.229765 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.230819 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.231106 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.231161 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.233578 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.233585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.233586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.233592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.235235 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.235631 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.235633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.235634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.235709 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.235909 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.235910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.235910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.236044 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.236103 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.236258 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.236339 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260927 07:03:54.236997 25685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:54.237339 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.237195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33106)
0927 07:03:54.237268 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.237316 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 07:03:54.237474 25635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
I20260927 07:03:54.238013 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.237891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33114)
0927 07:03:54.237950 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.237973 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 07:03:54.238107 25685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:38341: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
I20260927 07:03:54.238564 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:38341
--webserver_interface=127.3.57.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:54.260423 25610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } attempt: 3
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:54.267268 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.260750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48276)
0927 07:03:54.260823 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.260835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.260892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.261045 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.261046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.261173 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.261229 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.261230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.262114 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.262115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.262634 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.262636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.263097 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.263097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.263152 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.263612 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.263623 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.264726 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.266339 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.266340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.266340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.266518 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.266722 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.266722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.266722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.266771 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.267039 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.267039 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.267075 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.267127 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.267207 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 07:03:54.267345 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.260841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.3.57.61:43993)
0927 07:03:54.260915 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.260916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.260940 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:03:54.260963 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.260964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.260975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.261024 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.261305 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.262037 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.262717 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.263024 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.263091 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.264810 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.264811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.264811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.264818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.266262 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.266594 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.266595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.266595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.266653 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.266843 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.266844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.266844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.266972 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.267035 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.267186 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.267265 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:03:54.267722 25610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:54.268059 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.267946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33116)
0927 07:03:54.268007 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.268034 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 07:03:54.268347 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.268297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33124)
0927 07:03:54.268316 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.268334 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 07:03:54.268447 25610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:38341: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
I20260927 07:03:54.287348 25685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } attempt: 1
I20260927 07:03:54.287820 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.287696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33126)
0927 07:03:54.287756 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.287793 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 07:03:54.288228 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.288137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33132)
0927 07:03:54.288186 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.288211 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 07:03:54.288347 25685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:38341: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
I20260927 07:03:54.299599 25610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } attempt: 1
I20260927 07:03:54.299975 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.299855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33146)
0927 07:03:54.299913 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.299943 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:03:54.300288 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.300189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33158)
0927 07:03:54.300241 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.300266 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 07:03:54.300403 25610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:38341: Network error: Client connection negotiation failed: client connection to 127.3.57.60:38341: connect: Connection refused (error 111)
W20260927 07:03:54.319725 25692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:54.319942 25692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:54.319980 25692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:54.320019 25692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:54.321163 25692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:54.321224 25692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:54.321249 25692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:54.321276 25692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:54.321303 25692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:54.322851 25692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:44195,127.3.57.61:43993,127.3.57.60:38341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.57.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:54.323110 25692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:54.323324 25692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:54.328068 25698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:54.328184 25697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.328260 25692 server_base.cc:1034] running on GCE node
W20260927 07:03:54.328047 25700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:54.328542 25692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:54.328861 25692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:54.330055 25692 hybrid_clock.cc:648] HybridClock initialized: now 1790492634330043 us; error 26 us; skew 500 ppm
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:54.332621 25692 init.cc:377] Logged in from keytab as kudu/127.3.57.60@KRBTEST.COM (short username kudu)
I20260927 07:03:54.333025 25692 webserver.cc:460] Webserver started at http://127.3.57.60:33767/ using document root <none> and password file <none>
I20260927 07:03:54.333252 25692 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:54.333325 25692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:54.333480 25692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:54.334465 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data/instance:
uuid: "7708bd735ae840ac89ff121a087a2129"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "54727d95cf1315899809dcd8da600c6f"
server_key_iv: "60c73b6e07416202688998327139935f"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.334820 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal/instance:
uuid: "7708bd735ae840ac89ff121a087a2129"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "54727d95cf1315899809dcd8da600c6f"
server_key_iv: "60c73b6e07416202688998327139935f"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.336193 25692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:54.336933 25707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:54.337137 25692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 07:03:54.337239 25692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "7708bd735ae840ac89ff121a087a2129"
format_stamp: "Formatted at 2026-09-27 07:03:54 on dist-test-slave-mk3h"
server_key: "54727d95cf1315899809dcd8da600c6f"
server_key_iv: "60c73b6e07416202688998327139935f"
server_key_version: "encryptionkey@0"
I20260927 07:03:54.337320 25692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:54.347671 25692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:54.349946 25692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:54.350112 25692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:54.354218 25692 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.60:38341
I20260927 07:03:54.354279 25759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.60:38341 every 8 connection(s)
I20260927 07:03:54.354653 25692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
I20260927 07:03:54.355695 25760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:54.356491 25685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } attempt: 2
I20260927 07:03:54.357508 25760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:54.362938 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25692
I20260927 07:03:54.363031 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492341973781-3300-0/minicluster-data/master-2/wal/instance
I20260927 07:03:54.363224 3300 external_mini_cluster.cc:1428] Setting key 7e5857bfe5393fa3b223f6f2f04a2645
I20260927 07:03:54.366405 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.356767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33162)
0927 07:03:54.356830 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.356841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.356888 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.357717 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.357718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.357887 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.357931 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.357933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.359430 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.359431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.360223 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.360227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.360856 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.360857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.360908 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.361316 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.361320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.362590 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.365191 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.365192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.365407 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.365651 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.365652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.365714 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.366075 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.366076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.366121 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.366203 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.366302 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 07:03:54.366468 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.358145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.3.57.62:44195)
0927 07:03:54.358221 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.358224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.358330 (+ 106us) server_negotiation.cc:407] Connection header received
0927 07:03:54.358377 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.358378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.358388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.358460 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.358868 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.359590 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.360642 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.360925 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.360995 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.363430 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.363431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.363432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.363440 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.365139 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365548 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.365549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.365550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.365609 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365853 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.365854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.365854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.365981 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.366022 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.366266 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.366344 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 07:03:54.366676 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.356944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.3.57.60:38341)
0927 07:03:54.357109 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.357116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.357140 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:03:54.357523 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.357529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.357571 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.357720 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.358035 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.359334 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.360321 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.360774 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.360875 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.362685 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.362694 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.362694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.362702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.365100 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365497 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.365499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.365500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.365571 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365794 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.365795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.365795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.365991 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.366080 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.366258 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.366383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260927 07:03:54.366798 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.358063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:44195 (local address 127.0.0.1:51978)
0927 07:03:54.358225 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.358260 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.358331 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.358472 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.358473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.358696 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.358780 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.358783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.359657 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.359658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.360556 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.360558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.361035 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.361036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.361111 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.361453 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.361463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.363353 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.365218 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.365225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.365459 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.365680 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.365681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.365681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.365776 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.366066 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.366072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.366136 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.366552 (+ 416us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.366660 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260927 07:03:54.367368 25760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:54.368837 25685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: Bootstrap starting.
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:03:54.369536 25685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:54.369827 25685 log.cc:824] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:54.370733 25685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: No bootstrap required, opened a new log
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:54.373040 25685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.373216 25685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861a7b0bc2be4835b146db9842581351, State: Initialized, Role: FOLLOWER
I20260927 07:03:54.373327 25685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.373697 25685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:54.373715 25767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.373917 25767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.374187 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.363573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.3.57.60:38341)
0927 07:03:54.363740 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.363742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.363890 (+ 148us) server_negotiation.cc:407] Connection header received
0927 07:03:54.363964 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.363965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.363976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.364051 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.364809 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.365362 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.366455 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.366819 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.366888 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.369804 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.369806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.369806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.369813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.372433 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.372851 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.372852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.372852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.372913 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.373152 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.373153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.373154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.373290 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.373882 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.373935 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.374015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260927 07:03:54.375512 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.367626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:43993 (local address 127.0.0.1:48290)
0927 07:03:54.367766 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.367777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.367818 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.367951 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.367952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.368096 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.368159 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.368162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.369032 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.369033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.369824 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.369827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.370274 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.370275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.370450 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.371047 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.371051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.372484 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.374370 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.374370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.374371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.374581 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.374815 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.374816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.374816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.374882 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.375216 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.375216 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.375261 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.375325 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.375421 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 07:03:54.375609 25610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } attempt: 2
W20260927 07:03:54.375759 25778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:54.375800 25778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:54.375919 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.367739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.3.57.61:43993)
0927 07:03:54.367807 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.367809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.367817 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:54.367880 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.367881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.367890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.367944 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.368250 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.368940 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.369926 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.370183 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.370216 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.372593 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.372594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.372595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.372603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.374278 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.374663 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.374665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.374665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.374720 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.374959 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.374960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.374960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.375120 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.375186 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.375716 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.375791 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:03:54.376214 25760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:03:54.377409 25617 master_runner.cc:186] Error getting master registration for 127.3.57.62:44195: OK, instance_id { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" instance_seqno: 1790492634033686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:03:54.382313 25617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:54.385043 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.375900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33180)
0927 07:03:54.375945 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.375955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.375993 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.376191 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.376192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.376344 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.376383 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.376385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.377292 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.377292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.378046 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.378048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.378636 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.378637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.378707 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.379118 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.379123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.381172 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.383070 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.383071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.383072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.383314 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.383818 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.383819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.383819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.383889 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.384282 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.384283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.384330 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.384396 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.384918 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 07:03:54.385530 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.376426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:38341 (local address 127.0.0.1:33182)
0927 07:03:54.376477 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:54.376487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:54.376526 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:54.376660 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:54.376661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:54.376795 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:54.376822 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.376823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.377669 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.377670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.378365 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:54.378368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.379124 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.379125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:54.379179 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.379609 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:54.379614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:54.381709 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:54.384580 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.384580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.384581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.384777 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.384957 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:54.384957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:54.384958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:54.385008 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:54.385243 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:54.385243 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:54.385273 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 07:03:54.385334 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 07:03:54.385426 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:03:54.385782 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.375939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.3.57.60:38341)
0927 07:03:54.375974 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.375976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.375986 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:54.376074 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.376076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.376086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.376133 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.376480 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.377207 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.378146 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.378506 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.378578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.381273 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.381274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.381275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.381282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.382950 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.383400 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.383401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.383401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.383457 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.384025 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.384026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.384026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.384193 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.384255 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.384527 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.385338 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:54.385787 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.376451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.3.57.60:38341)
0927 07:03:54.376481 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.376483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.376514 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:03:54.376577 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.376578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.376586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.376631 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.376891 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.377587 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.378463 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.378745 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.378800 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.382315 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.382316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.382316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.382323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.384493 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.384850 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.384851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.384851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.384895 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.385066 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.385067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.385067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.385172 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.385223 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.385638 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.385709 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 07:03:54.387486 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: Bootstrap starting.
I20260927 07:03:54.387583 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.375120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.3.57.62:44195)
0927 07:03:54.375159 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.375161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.375168 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:03:54.375193 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.375194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.375205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.375253 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.376667 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.377342 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.380560 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.380883 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.380952 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.384374 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.384375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.384375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.384382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.386075 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.386492 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.386492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.386493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.386547 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.386743 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.386744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.386744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.386850 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.386976 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.387375 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.387466 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 07:03:54.387864 25610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: Bootstrap starting.
I20260927 07:03:54.388291 25760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:54.388356 25610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:54.388572 25610 log.cc:824] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:54.388574 25760 log.cc:824] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:54.389369 25610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: No bootstrap required, opened a new log
I20260927 07:03:54.389513 25760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: No bootstrap required, opened a new log
I20260927 07:03:54.391438 25760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.391592 25760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7708bd735ae840ac89ff121a087a2129, State: Initialized, Role: FOLLOWER
I20260927 07:03:54.391672 25760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.391480 25610 raft_consensus.cc:348] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.391737 25610 raft_consensus.cc:729] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edeee2b3e4ae4f43af44c59c3d40c286, State: Initialized, Role: FOLLOWER
I20260927 07:03:54.391811 25610 consensus_queue.cc:260] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.391945 25760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:54.392104 25610 sys_catalog.cc:564] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 07:03:54.394016 25799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:54.394091 25799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:54.394327 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.394472 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.394620 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.394739 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 07:03:54 dist-test-slave-mk3h krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:54.396622 25692 master_runner.cc:263] No leader master found from master 7708bd735ae840ac89ff121a087a2129
I20260927 07:03:54.396673 25692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:03:54.397089 25542 master_runner.cc:263] No leader master found from master edeee2b3e4ae4f43af44c59c3d40c286
I20260927 07:03:54.397132 25542 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:03:54.397986 25805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:54.398175 25805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:54.400560 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.389403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.3.57.61:43993)
0927 07:03:54.389459 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.389462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.389595 (+ 133us) server_negotiation.cc:407] Connection header received
0927 07:03:54.389616 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.389617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.389626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.389667 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.390263 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.390967 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.391919 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.392195 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.392272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.397183 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.397185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.397185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.397192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.399058 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.399495 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.399497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.399497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.399575 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.399784 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.399784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.399785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.399907 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.400036 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.400395 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.400469 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:54.402149 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:03:54.403203 25584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "edeee2b3e4ae4f43af44c59c3d40c286"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51990
I20260927 07:03:54.403328 25584 raft_consensus.cc:482] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:03:54.403386 25584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:54.404702 25584 raft_consensus.cc:504] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.404910 25584 leader_election.cc:288] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [CANDIDATE]: Term 1 election: Requested vote from peers 861a7b0bc2be4835b146db9842581351 (127.3.57.61:43993), 7708bd735ae840ac89ff121a087a2129 (127.3.57.60:38341)
I20260927 07:03:54.405272 25733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7708bd735ae840ac89ff121a087a2129"
I20260927 07:03:54.405396 25733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:54.406620 25733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edeee2b3e4ae4f43af44c59c3d40c286 in term 1.
I20260927 07:03:54.406802 25560 leader_election.cc:302] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7708bd735ae840ac89ff121a087a2129, edeee2b3e4ae4f43af44c59c3d40c286; no voters:
I20260927 07:03:54.407191 25658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "861a7b0bc2be4835b146db9842581351"
I20260927 07:03:54.407338 25658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:54.408322 25658 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edeee2b3e4ae4f43af44c59c3d40c286 in term 1.
I20260927 07:03:54.409341 25783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:54.409559 25783 raft_consensus.cc:686] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [term 1 LEADER]: Becoming Leader. State: Replica: edeee2b3e4ae4f43af44c59c3d40c286, State: Running, Role: LEADER
I20260927 07:03:54.409689 25783 consensus_queue.cc:237] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } }
I20260927 07:03:54.410192 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edeee2b3e4ae4f43af44c59c3d40c286. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.410324 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:54.410553 25814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:54.410772 25814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:54.413821 25733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [term 1 FOLLOWER]: Refusing update from remote peer edeee2b3e4ae4f43af44c59c3d40c286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:54.414057 25658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [term 1 FOLLOWER]: Refusing update from remote peer edeee2b3e4ae4f43af44c59c3d40c286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:54.414505 25783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:54.414638 25783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:54.415932 25767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edeee2b3e4ae4f43af44c59c3d40c286. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.416018 25767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.416983 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edeee2b3e4ae4f43af44c59c3d40c286. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.417065 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.425118 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.412076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.3.57.62:44195)
0927 07:03:54.412111 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.412113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.417310 (+ 5197us) server_negotiation.cc:407] Connection header received
0927 07:03:54.417425 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.417426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.417436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.417467 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.418235 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.418991 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.420065 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.420359 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.420398 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.422217 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.422219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.422219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.422242 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.423920 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.424296 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.424297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.424298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.424350 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.424555 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.424555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.424556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.424672 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.424759 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.424954 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.425028 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 07:03:54.428074 25812 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.428179 25812 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:54.428622 25767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.428735 25767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.429028 25815 mvcc.cc:205] Tried to move back new op lower bound from 7333857830556418048 to 7333857830543314944. Current Snapshot: MvccSnapshot[applied={T|T < 7333857830556418048}]
I20260927 07:03:54.429165 25819 mvcc.cc:205] Tried to move back new op lower bound from 7333857830556418048 to 7333857830543314944. Current Snapshot: MvccSnapshot[applied={T|T < 7333857830556418048}]
I20260927 07:03:54.429675 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.429749 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:54.430176 25782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeee2b3e4ae4f43af44c59c3d40c286" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 44195 } } peers { permanent_uuid: "861a7b0bc2be4835b146db9842581351" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 43993 } } peers { permanent_uuid: "7708bd735ae840ac89ff121a087a2129" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 38341 } } }
I20260927 07:03:54.430254 25782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:54.430511 25821 mvcc.cc:205] Tried to move back new op lower bound from 7333857830556418048 to 7333857830543314944. Current Snapshot: MvccSnapshot[applied={T|T < 7333857830556418048}]
I20260927 07:03:54.431380 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.412446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.3.57.60:38341)
0927 07:03:54.412483 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.412485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.419698 (+ 7213us) server_negotiation.cc:407] Connection header received
0927 07:03:54.419798 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.419799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.419809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.419836 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.420574 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.421271 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.422276 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.422563 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.422601 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.428502 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.428503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.428504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.428511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.430178 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.430571 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.430572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.430572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.430626 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.430815 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.430816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.430816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.430932 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.431012 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.431220 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.431294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:03:54.435207 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:54.412761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.3.57.61:43993)
0927 07:03:54.412797 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.412798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.423314 (+ 10516us) server_negotiation.cc:407] Connection header received
0927 07:03:54.423411 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.423412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.423421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.423449 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.424059 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.424771 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.425756 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:54.426069 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:54.426106 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:54.432258 (+ 6152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:54.432259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:54.432259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:54.432267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:54.433913 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.434310 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.434311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.434312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.434365 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:54.434611 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:54.434612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:54.434613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:54.434767 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:54.434847 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:54.435040 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 07:03:54.435116 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:54.436416 25814 catalog_manager.cc:1347] Generated new cluster ID: 818ff4ce1a764b218fb858a8ed416ac4
I20260927 07:03:54.436470 25814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:54.478292 25814 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:54.480952 25814 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:54.494359 25814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edeee2b3e4ae4f43af44c59c3d40c286: Generated new TSK 0
I20260927 07:03:54.494700 25814 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:03:54.522923 25613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.520832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.3.57.62:44195)
0927 07:03:54.520887 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.520890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.522096 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 07:03:54.522231 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.522233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.522246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.522280 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.522830 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52014: BlockingRecv error: recv got EOF from 127.0.0.1:52014 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 07:03:54.524003 25764 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.521308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.3.57.60:38341)
0927 07:03:54.521370 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.521372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.523381 (+ 2009us) server_negotiation.cc:407] Connection header received
0927 07:03:54.523490 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.523491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.523503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.523536 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.523935 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33202: BlockingRecv error: recv got EOF from 127.0.0.1:33202 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 07:03:54.524983 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.521720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.3.57.61:43993)
0927 07:03:54.521764 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.521766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.524400 (+ 2634us) server_negotiation.cc:407] Connection header received
0927 07:03:54.524518 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.524519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.524532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.524566 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.524895 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48308: BlockingRecv error: recv got EOF from 127.0.0.1:48308 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 07:03:54.527976 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.527006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.3.57.61:43993)
0927 07:03:54.527049 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.527051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.527327 (+ 276us) server_negotiation.cc:407] Connection header received
0927 07:03:54.527450 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.527451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.527474 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.527507 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.527924 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48318: BlockingRecv error: recv got EOF from 127.0.0.1:48318 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:03:54.531375 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.530508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.3.57.61:43993)
0927 07:03:54.530545 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.530547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.530787 (+ 240us) server_negotiation.cc:407] Connection header received
0927 07:03:54.530892 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.530893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.530929 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.530962 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.531320 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48328: BlockingRecv error: recv got EOF from 127.0.0.1:48328 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:54.535689 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.534781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.3.57.61:43993)
0927 07:03:54.534819 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.534821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.535085 (+ 264us) server_negotiation.cc:407] Connection header received
0927 07:03:54.535190 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.535191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.535201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.535233 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.535637 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48336: BlockingRecv error: recv got EOF from 127.0.0.1:48336 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:54.542008 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.541176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.3.57.61:43993)
0927 07:03:54.541211 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.541213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.541457 (+ 244us) server_negotiation.cc:407] Connection header received
0927 07:03:54.541564 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.541565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.541575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.541606 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.541962 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48350: BlockingRecv error: recv got EOF from 127.0.0.1:48350 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:54.552340 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.551384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.3.57.61:43993)
0927 07:03:54.551425 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.551427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.551723 (+ 296us) server_negotiation.cc:407] Connection header received
0927 07:03:54.551843 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.551844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.551856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.551887 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.552272 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48358: BlockingRecv error: recv got EOF from 127.0.0.1:48358 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 07:03:54.570976 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.569946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.3.57.61:43993)
0927 07:03:54.569992 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.569994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.570294 (+ 300us) server_negotiation.cc:407] Connection header received
0927 07:03:54.570429 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.570430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.570442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.570475 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.570913 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48366: BlockingRecv error: recv got EOF from 127.0.0.1:48366 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 07:03:54.605268 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.604524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.3.57.61:43993)
0927 07:03:54.604597 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.604600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.604712 (+ 112us) server_negotiation.cc:407] Connection header received
0927 07:03:54.604787 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.604788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.604801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.604858 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.605209 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48368: BlockingRecv error: recv got EOF from 127.0.0.1:48368 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 07:03:54.671308 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.670612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.3.57.61:43993)
0927 07:03:54.670688 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.670690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.670777 (+ 87us) server_negotiation.cc:407] Connection header received
0927 07:03:54.670829 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.670830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.670840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.670904 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.671236 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48374: BlockingRecv error: recv got EOF from 127.0.0.1:48374 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 07:03:54.801190 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:54.800532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.3.57.61:43993)
0927 07:03:54.800587 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:54.800589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:54.800686 (+ 97us) server_negotiation.cc:407] Connection header received
0927 07:03:54.800745 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:54.800746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:54.800756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:54.800819 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:54.801118 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48376: BlockingRecv error: recv got EOF from 127.0.0.1:48376 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 07:03:55.059091 25690 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:55.058494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.3.57.61:43993)
0927 07:03:55.058583 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.058585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.058593 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:55.058619 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.058620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.058629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.058688 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:55.059009 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48380: BlockingRecv error: recv got EOF from 127.0.0.1:48380 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:03:55.376230 25778 catalog_manager.cc:1259] Loaded cluster ID: 818ff4ce1a764b218fb858a8ed416ac4
I20260927 07:03:55.376335 25778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: loading cluster ID for follower catalog manager: success
I20260927 07:03:55.377154 25778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: acquiring CA information for follower catalog manager: success
I20260927 07:03:55.377457 25778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 861a7b0bc2be4835b146db9842581351: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:03:55.394488 25799 catalog_manager.cc:1259] Loaded cluster ID: 818ff4ce1a764b218fb858a8ed416ac4
I20260927 07:03:55.394544 25799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: loading cluster ID for follower catalog manager: success
I20260927 07:03:55.395426 25799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: acquiring CA information for follower catalog manager: success
I20260927 07:03:55.395745 25799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7708bd735ae840ac89ff121a087a2129: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:03:55.574920 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:55.572271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.3.57.61:43993)
0927 07:03:55.572377 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.572379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.572404 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:03:55.572471 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.572472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.572534 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.572600 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:55.572739 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.573237 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.574029 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.574234 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.574301 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:55.574498 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:55.574654 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:55.574752 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:55.574771 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 07:03:55.574797 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260927 07:03:55.579044 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:55.576703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.3.57.60:38341)
0927 07:03:55.576838 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.576843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.576851 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:55.576891 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.576891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.576923 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.577008 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:55.577097 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.577567 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.578294 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.578489 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.578566 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:55.578717 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:55.578810 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:55.578894 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:55.578935 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 07:03:55.578958 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260927 07:03:55.583530 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:55.580663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.3.57.61:43993)
0927 07:03:55.580701 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.580703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.580811 (+ 108us) server_negotiation.cc:407] Connection header received
0927 07:03:55.580869 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.580870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.580880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.580968 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:55.581053 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.581768 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.582783 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.583032 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.583095 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:55.583282 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:55.583314 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:55.583376 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:55.583435 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 07:03:55.583457 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:55.583966 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:55.580737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.3.57.60:38341)
0927 07:03:55.580777 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.580779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.581309 (+ 530us) server_negotiation.cc:407] Connection header received
0927 07:03:55.581366 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.581367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.581375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.581444 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:55.581564 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.582157 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.583053 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.583350 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.583413 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:55.583713 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:55.583742 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:55.583768 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:55.583871 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 07:03:55.583893 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:55.584062 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:55.580654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.3.57.62:44195)
0927 07:03:55.581051 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:55.581052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:55.581058 (+ 6us) server_negotiation.cc:407] Connection header received
0927 07:03:55.581089 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:55.581090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:55.581122 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:55.581179 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:55.581339 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.582023 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.583129 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:55.583466 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:55.583524 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:55.583737 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:55.583834 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:55.583891 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:55.583949 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 07:03:55.583972 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":46,"threads_started":1}
I20260927 07:03:55.585553 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25542
I20260927 07:03:55.591128 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25617
I20260927 07:03:55.595935 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25692
2026-09-27T07:03:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2834 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:03:55 dist-test-slave-mk3h krb5kdc[25907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:03:55 dist-test-slave-mk3h krb5kdc[25907](info): set up 2 sockets
Sep 27 07:03:55 dist-test-slave-mk3h krb5kdc[25907](info): commencing operation
krb5kdc: starting...
W20260927 07:03:56.631762 3300 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/test-user.keytab.
Sep 27 07:03:56 dist-test-slave-mk3h krb5kdc[25907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:03:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:03:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.57.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
Entry for principal kudu/127.3.57.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab.
I20260927 07:03:56.760597 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40107
--webserver_interface=127.3.57.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:03:56.839432 25920 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:56.839767 25920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:56.839807 25920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:56.839843 25920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:56.841039 25920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:56.841113 25920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:56.841140 25920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:56.841166 25920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:56.841192 25920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:56.842653 25920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.62.keytab
--principal=kudu/127.3.57.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.62:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.57.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:56.842936 25920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:56.843194 25920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:56.848016 25926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:56.848057 25928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:56.848166 25925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:56.848088 25920 server_base.cc:1034] running on GCE node
I20260927 07:03:56.848455 25920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:56.848757 25920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:56.849933 25920 hybrid_clock.cc:648] HybridClock initialized: now 1790492636849923 us; error 51 us; skew 500 ppm
Sep 27 07:03:56 dist-test-slave-mk3h krb5kdc[25907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:56.852291 25920 init.cc:377] Logged in from keytab as kudu/127.3.57.62@KRBTEST.COM (short username kudu)
I20260927 07:03:56.852734 25920 webserver.cc:460] Webserver started at http://127.3.57.62:32975/ using document root <none> and password file <none>
I20260927 07:03:56.852907 25920 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:56.852953 25920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:56.853039 25920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:56.853992 25920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data/instance:
uuid: "47a8420e9666435ebfcba0b91164d8a7"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "e91a3719cc4ec0c0af3543ee949c66de"
server_key_iv: "764d1da5ea8b0b7b7dc5ea18144b6990"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.854278 25920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance:
uuid: "47a8420e9666435ebfcba0b91164d8a7"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "e91a3719cc4ec0c0af3543ee949c66de"
server_key_iv: "764d1da5ea8b0b7b7dc5ea18144b6990"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.855670 25920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 07:03:56.856354 25935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:56.856577 25920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:56.856660 25920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal
uuid: "47a8420e9666435ebfcba0b91164d8a7"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "e91a3719cc4ec0c0af3543ee949c66de"
server_key_iv: "764d1da5ea8b0b7b7dc5ea18144b6990"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.856715 25920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:56.863086 25920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:56.865218 25920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:56.865319 25920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:56.869060 25920 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.62:40107
I20260927 07:03:56.869109 25987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.62:40107 every 8 connection(s)
I20260927 07:03:56.869431 25920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/data/info.pb
I20260927 07:03:56.870401 25988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:56.872002 25988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:56.874912 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25920
I20260927 07:03:56.875022 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-0/wal/instance
I20260927 07:03:56.875200 3300 external_mini_cluster.cc:1428] Setting key c3301d33e664eaea851f69c4beb64cf4
Sep 27 07:03:56 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.61@KRBTEST.COM; defaulting to no policy
I20260927 07:03:56.882613 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.872779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.3.57.62:40107)
0927 07:03:56.872905 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:56.872910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:56.872941 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:03:56.873485 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:56.873487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:56.873506 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:56.873619 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:56.874160 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:56.875213 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:56.876154 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:56.876667 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:56.876812 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:56.879242 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:56.879250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:56.879250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:56.879257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:56.880896 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:56.881315 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:56.881318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:56.881318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:56.881400 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:56.881663 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:56.881664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:56.881664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:56.881867 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:56.881968 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:56.882159 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 07:03:56.882334 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":93,"thread_start_us":40,"threads_started":1}
I20260927 07:03:56.882615 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.872638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40107 (local address 127.0.0.1:51436)
0927 07:03:56.872779 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:56.872811 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:56.873309 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:56.873624 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:56.873626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:56.873889 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:56.874044 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:56.874048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:56.875323 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:56.875324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:56.876059 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:56.876061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:56.876798 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:56.876799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:56.876863 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:56.877391 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:56.877399 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:56.879125 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:56.881015 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:56.881021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:56.881023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:56.881213 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:56.881498 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:56.881499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:56.881499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:56.881589 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:56.881952 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:56.881957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:56.882018 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 07:03:56.882133 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 07:03:56.882372 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":103,"thread_start_us":62,"threads_started":1}
I20260927 07:03:56.883324 25988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:56.883771 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.883650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50916)
0927 07:03:56.883702 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:56.883750 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 07:03:56.883916 25938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
I20260927 07:03:56.884290 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.884189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50922)
0927 07:03:56.884243 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:56.884266 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 07:03:56.884394 25988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:40681: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
Entry for principal kudu/127.3.57.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab.
I20260927 07:03:56.900795 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:40681
--webserver_interface=127.3.57.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:56.933645 25988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } attempt: 1
I20260927 07:03:56.934286 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.934116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50924)
0927 07:03:56.934207 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:56.934255 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 07:03:56.934653 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:56.934559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50926)
0927 07:03:56.934606 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:56.934630 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:03:56.934738 25988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:40681: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
W20260927 07:03:56.980080 25995 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:56.980312 25995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:56.980341 25995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:56.980378 25995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:56.981451 25995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:56.981519 25995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:56.981545 25995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:56.981572 25995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:56.981591 25995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:56.982977 25995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.61.keytab
--principal=kudu/127.3.57.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.61:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.57.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:56.983260 25995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:56.983484 25995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:03:56.988281 25995 server_base.cc:1034] running on GCE node
W20260927 07:03:56.988261 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:56.988247 26000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:56.988262 26003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:56.988829 25995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:56.989075 25995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:56.990213 25995 hybrid_clock.cc:648] HybridClock initialized: now 1790492636990200 us; error 28 us; skew 500 ppm
Sep 27 07:03:56 dist-test-slave-mk3h krb5kdc[25907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:56.992602 25995 init.cc:377] Logged in from keytab as kudu/127.3.57.61@KRBTEST.COM (short username kudu)
I20260927 07:03:56.993077 25995 webserver.cc:460] Webserver started at http://127.3.57.61:32963/ using document root <none> and password file <none>
I20260927 07:03:56.993299 25995 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:56.993371 25995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:56.993525 25995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:56.994529 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data/instance:
uuid: "b2934beb1b5e45eaa7a42b449dd44783"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "0a95be9ff30d6163c84174bef07467e7"
server_key_iv: "dba7de988579d6edb210654a8f751c50"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.994889 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal/instance:
uuid: "b2934beb1b5e45eaa7a42b449dd44783"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "0a95be9ff30d6163c84174bef07467e7"
server_key_iv: "dba7de988579d6edb210654a8f751c50"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.996301 25995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 07:03:56.997067 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:56.997259 25995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 07:03:56.997360 25995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal
uuid: "b2934beb1b5e45eaa7a42b449dd44783"
format_stamp: "Formatted at 2026-09-27 07:03:56 on dist-test-slave-mk3h"
server_key: "0a95be9ff30d6163c84174bef07467e7"
server_key_iv: "dba7de988579d6edb210654a8f751c50"
server_key_version: "encryptionkey@0"
I20260927 07:03:56.997440 25995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:57.002871 25988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } attempt: 2
I20260927 07:03:57.003216 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.003102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50942)
0927 07:03:57.003159 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.003190 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 07:03:57.003533 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.003441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50944)
0927 07:03:57.003487 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.003510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:03:57.003687 25988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.61:40681: Network error: Client connection negotiation failed: client connection to 127.3.57.61:40681: connect: Connection refused (error 111)
I20260927 07:03:57.009341 25995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:57.010970 25995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:57.011073 25995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:57.014961 25995 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.61:40681
I20260927 07:03:57.015002 26062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.61:40681 every 8 connection(s)
I20260927 07:03:57.015311 25995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/data/info.pb
I20260927 07:03:57.016213 26063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:57.017676 26063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:57.025202 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 25995
I20260927 07:03:57.025301 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-1/wal/instance
I20260927 07:03:57.025529 3300 external_mini_cluster.cc:1428] Setting key 20bf94b5d9274b49e26b5e94da5e4dcd
I20260927 07:03:57.026243 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.018341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.3.57.62:40107)
0927 07:03:57.018405 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.018407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.018649 (+ 242us) server_negotiation.cc:407] Connection header received
0927 07:03:57.019081 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.019082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.019092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.019144 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.019746 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.020221 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.021140 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.021323 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.021375 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.023399 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.023400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.023401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.023407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.024670 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.025085 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.025086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.025087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.025152 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.025492 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.025493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.025494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.025653 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.025731 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.026039 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.026128 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:03:57.026393 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.018337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40107 (local address 127.0.0.1:51440)
0927 07:03:57.018521 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.018553 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.019002 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.019217 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.019220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.019442 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.019646 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.019650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.020305 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.020306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.021061 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.021063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.021406 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.021407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.021502 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.021998 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.022012 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.023296 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.024739 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.024745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.024747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.025000 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.025226 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.025227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.025227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.025404 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.025734 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.025741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.025874 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.025988 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.026160 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260927 07:03:57.027027 26063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.57.60@KRBTEST.COM; defaulting to no policy
I20260927 07:03:57.034993 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.027259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50956)
0927 07:03:57.027307 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.027317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.027356 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.027640 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.027641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.027751 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.027797 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.027798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.029042 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.029042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.029510 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.029511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.030164 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.030165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.030213 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.030614 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.030619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.031956 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.033914 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.033915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.033915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.034092 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.034317 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.034317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.034318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.034371 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.034664 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.034665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.034700 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.034761 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.034853 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 07:03:57.035069 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.027315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.3.57.61:40681)
0927 07:03:57.027469 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.027474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.027497 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:03:57.027521 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.027523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.027538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.027638 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.027873 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.028966 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.029589 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.030088 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.030164 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.032047 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.032055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.032055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.032062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.033806 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.034190 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.034193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.034193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.034247 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.034451 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.034452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.034452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.034584 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.034671 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.034787 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.034920 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 07:03:57.035696 26063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:57.036000 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.035894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40634)
0927 07:03:57.035940 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.035982 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:03:57.036139 26012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
I20260927 07:03:57.036514 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.036427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40636)
0927 07:03:57.036475 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.036498 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 07:03:57.036634 26063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:42325: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.57.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
Entry for principal kudu/127.3.57.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab.
I20260927 07:03:57.051156 3300 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskaZGoNp/build/release/bin/kudu
/tmp/dist-test-taskaZGoNp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:42325
--webserver_interface=127.3.57.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.57.20:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:03:57.085878 26063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } attempt: 1
I20260927 07:03:57.086378 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.086246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40646)
0927 07:03:57.086311 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.086349 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:03:57.086721 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.086624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40662)
0927 07:03:57.086673 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.086698 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 07:03:57.086836 26063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:42325: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
I20260927 07:03:57.094858 25988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } attempt: 3
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:57.102313 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.095158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50964)
0927 07:03:57.095216 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.095227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.095283 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.095459 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.095460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.095677 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.095760 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.095762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.096455 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.096456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.096985 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.096986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.097383 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.097384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.097468 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.097954 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.097959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.099330 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.101216 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.101216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.101217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.101387 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.101615 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.101615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.101616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.101675 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.101966 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.101967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.102014 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.102110 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.102212 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:03:57.102312 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.095224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.3.57.61:40681)
0927 07:03:57.095267 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.095269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.095282 (+ 13us) server_negotiation.cc:407] Connection header received
0927 07:03:57.095368 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.095369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.095387 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.095454 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.095849 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.096354 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.097080 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.097303 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.097388 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.099428 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.099429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.099430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.099437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.101114 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.101481 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.101483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.101483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.101544 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.101777 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.101778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.101778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.101895 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.101974 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.102133 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.102212 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 07:03:57.102846 25988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:57.103194 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.103102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40676)
0927 07:03:57.103143 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.103172 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 07:03:57.103525 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.103470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40682)
0927 07:03:57.103494 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.103513 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 07:03:57.103641 25988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:42325: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
I20260927 07:03:57.134814 25988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } attempt: 1
I20260927 07:03:57.135434 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.135221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40688)
0927 07:03:57.135359 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.135401 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 07:03:57.136020 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.135820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40692)
0927 07:03:57.135943 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.135990 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 07:03:57.136148 25988 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:42325: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
W20260927 07:03:57.138019 26070 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:03:57.138245 26070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:03:57.138266 26070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:03:57.138290 26070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:03:57.139518 26070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:03:57.139616 26070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:03:57.139652 26070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:03:57.139667 26070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:03:57.139680 26070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:03:57.141412 26070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.57.20:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.57.62:40107,127.3.57.61:40681,127.3.57.60:42325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/krb5kdc/kudu_127.3.57.60.keytab
--principal=kudu/127.3.57.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.57.60:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.57.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 06:08:55 UTC on ccycloud.quasar-ezlzfr.root.comops.site
build id 4675736
I20260927 07:03:57.141673 26070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:03:57.141856 26070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:03:57.148128 26078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:57.148128 26076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:03:57.148883 26075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:03:57.149350 26070 server_base.cc:1034] running on GCE node
I20260927 07:03:57.149801 26070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:03:57.150143 26070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:03:57.151363 26070 hybrid_clock.cc:648] HybridClock initialized: now 1790492637151344 us; error 44 us; skew 500 ppm
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:03:57.154328 26070 init.cc:377] Logged in from keytab as kudu/127.3.57.60@KRBTEST.COM (short username kudu)
I20260927 07:03:57.154815 26070 webserver.cc:460] Webserver started at http://127.3.57.60:41389/ using document root <none> and password file <none>
I20260927 07:03:57.154985 26063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } attempt: 2
I20260927 07:03:57.155114 26070 fs_manager.cc:359] Metadata directory not provided
I20260927 07:03:57.155174 26070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:03:57.155354 26070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:03:57.155544 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.155270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40702)
0927 07:03:57.155335 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.155377 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 07:03:57.156697 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data/instance:
uuid: "7012bc2239f445fa8b7c8686b8a0d777"
format_stamp: "Formatted at 2026-09-27 07:03:57 on dist-test-slave-mk3h"
server_key: "492de66040e47c90ab548dfe24124e03"
server_key_iv: "87fc342f1aaac2d707a56f5b4166a17a"
server_key_version: "encryptionkey@0"
I20260927 07:03:57.156868 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.156704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40710)
0927 07:03:57.156801 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.156832 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 07:03:57.157004 26063 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.57.60:42325: Network error: Client connection negotiation failed: client connection to 127.3.57.60:42325: connect: Connection refused (error 111)
I20260927 07:03:57.157136 26070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal/instance:
uuid: "7012bc2239f445fa8b7c8686b8a0d777"
format_stamp: "Formatted at 2026-09-27 07:03:57 on dist-test-slave-mk3h"
server_key: "492de66040e47c90ab548dfe24124e03"
server_key_iv: "87fc342f1aaac2d707a56f5b4166a17a"
server_key_version: "encryptionkey@0"
I20260927 07:03:57.158674 26070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 07:03:57.159868 26085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:03:57.160388 26070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 07:03:57.160770 26070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data,/tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal
uuid: "7012bc2239f445fa8b7c8686b8a0d777"
format_stamp: "Formatted at 2026-09-27 07:03:57 on dist-test-slave-mk3h"
server_key: "492de66040e47c90ab548dfe24124e03"
server_key_iv: "87fc342f1aaac2d707a56f5b4166a17a"
server_key_version: "encryptionkey@0"
I20260927 07:03:57.160884 26070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:03:57.180779 26070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:03:57.186296 26070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:03:57.186429 26070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:03:57.191696 26070 rpc_server.cc:307] RPC server started. Bound to: 127.3.57.60:42325
I20260927 07:03:57.191869 26137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.57.60:42325 every 8 connection(s)
I20260927 07:03:57.192166 26070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/data/info.pb
I20260927 07:03:57.193210 26138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:03:57.194818 26138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:57.196367 3300 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskaZGoNp/build/release/bin/kudu as pid 26070
I20260927 07:03:57.196525 3300 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskaZGoNp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492341973781-3300-0/minicluster-data/master-2/wal/instance
I20260927 07:03:57.196755 3300 external_mini_cluster.cc:1428] Setting key 6307cc4a6ace56ba817ea7d40e386429
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
I20260927 07:03:57.211342 25988 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } attempt: 2
I20260927 07:03:57.212074 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.197185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.3.57.60:42325)
0927 07:03:57.197384 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.197388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.197409 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:03:57.197465 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.197467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.197492 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.197549 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.198310 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.199528 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.200645 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.200982 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.201080 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.205948 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.205960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.205960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.205987 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.209534 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.210251 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.210255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.210256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.210423 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.210705 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.210706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.210707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.211104 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.211275 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.211583 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.211778 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":166,"thread_start_us":133,"threads_started":1}
I20260927 07:03:57.212682 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.195816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.3.57.62:40107)
0927 07:03:57.195887 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.195889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.195900 (+ 11us) server_negotiation.cc:407] Connection header received
0927 07:03:57.196395 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.196396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.196410 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.196492 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.197396 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.198213 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.199630 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.199931 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.200007 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.208665 (+ 8658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.208668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.208669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.208679 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.211017 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.211587 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.211589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.211589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.211652 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.212012 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.212014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.212014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.212205 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.212287 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.212516 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.212594 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:03:57.213004 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.195601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.62:40107 (local address 127.0.0.1:51442)
0927 07:03:57.195787 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.195827 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.196265 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.196593 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.196597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.196962 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.197271 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.197278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.198310 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.198311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.199490 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.199493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.200037 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.200038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.200185 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.203621 (+ 3436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.203633 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.208341 (+ 4708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.211114 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.211121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.211123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.211449 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.211835 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.211835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.211836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.211904 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.212294 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.212301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.212363 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.212765 (+ 402us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.212889 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":20,"thread_start_us":74,"threads_started":1}
I20260927 07:03:57.214246 26138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.62@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.62@KRBTEST.COM
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:57.220410 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.211654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40722)
0927 07:03:57.211721 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.211732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.211764 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.212147 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.212148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.212332 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.212381 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.212383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.213271 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.213272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.213980 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.213982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.214470 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.214471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.214536 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.215022 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.215027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.216524 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.218615 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.218616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.218616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.218809 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.219024 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.219025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.219025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.219103 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.219747 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.219748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.220130 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.220191 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.220287 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 07:03:57.221324 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.211789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.3.57.60:42325)
0927 07:03:57.212015 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.212017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.212029 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:57.212044 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.212045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.212057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.212139 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.212478 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.213171 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.214065 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.214361 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.214692 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.216614 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.216616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.216616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.216622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.218372 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.218895 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.218896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.218897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.218961 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.219468 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.219469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.219470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.219642 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.221045 (+ 1403us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.221091 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.221174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":134,"threads_started":1}
I20260927 07:03:57.222398 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.213363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.3.57.62:40107)
0927 07:03:57.213409 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.213411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.213419 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:57.213467 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.213468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.213476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.213530 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.214409 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.215201 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.216053 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.216300 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.216365 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.218956 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.218957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.218957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.218963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.220697 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.221116 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.221116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.221117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.221174 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.221401 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.221402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.221402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.221685 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.221853 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.222256 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.222325 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:03:57.223532 25988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: Bootstrap starting.
I20260927 07:03:57.223658 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.214586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.3.57.61:40681)
0927 07:03:57.214631 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.214633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.214640 (+ 7us) server_negotiation.cc:407] Connection header received
0927 07:03:57.214664 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.214665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.214674 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.214726 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.215202 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.215971 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.216944 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.217247 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.217314 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.220362 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.220363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.220364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.220371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.222073 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.222448 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.222449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.222450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.222506 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.222736 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.222737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.222738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.223002 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.223090 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.223454 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.223536 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 07:03:57.223658 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.214499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.61:40681 (local address 127.0.0.1:50974)
0927 07:03:57.214537 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.214546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.214579 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.214771 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.214772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.215038 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.215097 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.215100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.216073 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.216074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.216846 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.216848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.217370 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.217371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.217449 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.218536 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.218542 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.220264 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.222159 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.222160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.222161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.222370 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.222582 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.222583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.222584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.222648 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.223102 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.223103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.223153 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.223419 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.223536 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 07:03:57.223986 26138 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:03:57.225662 25988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:57.226286 25988 log.cc:824] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.60@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:03:57.229239 25988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: No bootstrap required, opened a new log
I20260927 07:03:57.231874 25988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.232012 25988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a8420e9666435ebfcba0b91164d8a7, State: Initialized, Role: FOLLOWER
I20260927 07:03:57.232077 25988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.232548 25988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:57.232502 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.232813 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.233382 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.224317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.3.57.60:42325)
0927 07:03:57.224348 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.224350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.224382 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:03:57.224449 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.224450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.224459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.224514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.224762 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.225248 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.225944 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.226169 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.226263 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.229182 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.229183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.229184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.229191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.232050 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.232529 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.232531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.232531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.232587 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.232778 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.232779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.232779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.232914 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.232955 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.233161 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.233253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.57.61@KRBTEST.COM
I20260927 07:03:57.233728 26140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.224277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40734)
0927 07:03:57.224344 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.224355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.224397 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.224518 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.224518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.224652 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.224682 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.224683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.225352 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.225353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.225843 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.225844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.226262 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.226263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.226311 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.227639 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.227645 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.229085 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.232190 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.232192 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.232192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.232419 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.232655 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.232655 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.232656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.232706 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.232999 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.233000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.233039 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.233520 (+ 481us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.233613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 07:03:57.234897 26157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:57.234961 26157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:57.236667 26138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: Bootstrap starting.
I20260927 07:03:57.237496 26138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:57.237811 26138 log.cc:824] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:57.238565 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.223246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50986 (local address 127.3.57.61:40681)
0927 07:03:57.223474 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.223476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.223486 (+ 10us) server_negotiation.cc:407] Connection header received
0927 07:03:57.223508 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.223508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.223519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.225550 (+ 2031us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.225619 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.228173 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.229484 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.229813 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.229891 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.234401 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.234404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.234404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.234415 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.236687 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.237189 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.237190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.237191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.237276 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.237489 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.237490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.237490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.237643 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.237806 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.238311 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.238412 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
I20260927 07:03:57.238793 26138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: No bootstrap required, opened a new log
I20260927 07:03:57.236625 25920 master_runner.cc:186] Error getting master registration for 127.3.57.61:40681: OK, instance_id { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" instance_seqno: 1790492637013836 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:03:57.239662 25920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:03:57.240753 26138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.240922 26138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7012bc2239f445fa8b7c8686b8a0d777, State: Initialized, Role: FOLLOWER
I20260927 07:03:57.241016 26138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.241308 26160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.241437 26160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.241380 26138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:57.242715 3300 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 07:03:57.244010 26171 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 07:03:57.243959 25962 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "47a8420e9666435ebfcba0b91164d8a7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51454
I20260927 07:03:57.244091 25962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:03:57.244158 25962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 0 FOLLOWER]: Advancing to term 1
W20260927 07:03:57.244232 26171 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:57.245102 25962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.245388 25962 leader_election.cc:288] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [CANDIDATE]: Term 1 election: Requested vote from peers b2934beb1b5e45eaa7a42b449dd44783 (127.3.57.61:40681), 7012bc2239f445fa8b7c8686b8a0d777 (127.3.57.60:42325)
I20260927 07:03:57.245697 26112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47a8420e9666435ebfcba0b91164d8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7012bc2239f445fa8b7c8686b8a0d777"
I20260927 07:03:57.245821 26112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:57.246901 26112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8420e9666435ebfcba0b91164d8a7 in term 1.
I20260927 07:03:57.247130 25937 leader_election.cc:302] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a8420e9666435ebfcba0b91164d8a7, 7012bc2239f445fa8b7c8686b8a0d777; no voters:
I20260927 07:03:57.247268 26146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:03:57.247430 26146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 47a8420e9666435ebfcba0b91164d8a7, State: Running, Role: LEADER
I20260927 07:03:57.247531 26146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.248066 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a8420e9666435ebfcba0b91164d8a7. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.248174 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:57.248288 26063 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } attempt: 3
I20260927 07:03:57.247193 26070 master_runner.cc:186] Error getting master registration for 127.3.57.61:40681: OK, instance_id { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" instance_seqno: 1790492637013836 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:03:57.251215 26070 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 07:03:57 dist-test-slave-mk3h krb5kdc[25907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.57.61@KRBTEST.COM for kudu/127.3.57.60@KRBTEST.COM
I20260927 07:03:57.258288 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.248499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.57.60:42325 (local address 127.0.0.1:40756)
0927 07:03:57.248571 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 07:03:57.248582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 07:03:57.248619 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:03:57.248946 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:03:57.248947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:03:57.249141 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:03:57.249194 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.249196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.250631 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.250632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.251439 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:03:57.251441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.252158 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.252160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:03:57.252216 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.252768 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:03:57.252773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:03:57.254375 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:03:57.256950 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.256952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.256952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.257192 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.257486 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:03:57.257487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:03:57.257487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:03:57.257561 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:03:57.257966 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:03:57.257967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:03:57.258028 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 07:03:57.258078 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 07:03:57.258184 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 07:03:57.259089 26182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:03:57.259258 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.248687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.3.57.60:42325)
0927 07:03:57.248790 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.248792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.248800 (+ 8us) server_negotiation.cc:407] Connection header received
0927 07:03:57.248822 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.248822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.248845 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.248900 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.249282 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.250520 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.251695 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.252034 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.252101 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.254486 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.254488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.254488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.254495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.256831 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.257293 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.257294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.257295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.257380 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.257643 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.257644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.257645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.257880 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.258830 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.258908 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.259042 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:03:57.259367 26182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:03:57.260288 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.246770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.3.57.62:40107)
0927 07:03:57.246826 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.246828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.251816 (+ 4988us) server_negotiation.cc:407] Connection header received
0927 07:03:57.251931 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.251932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.251944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.251976 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.253021 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.253713 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.255032 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.255330 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.255375 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.257326 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.257328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.257328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.257336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.258984 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.259387 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.259388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.259389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.259446 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.259682 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.259683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.259684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.259810 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.259878 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.260115 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.260199 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 07:03:57.261411 26063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: Bootstrap starting.
I20260927 07:03:57.262187 26063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: Neither blocks nor log segments found. Creating new log.
I20260927 07:03:57.262496 26063 log.cc:824] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: Log is configured to *not* fsync() on all Append() calls
I20260927 07:03:57.264499 26063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: No bootstrap required, opened a new log
I20260927 07:03:57.266882 26063 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.267050 26063 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2934beb1b5e45eaa7a42b449dd44783, State: Initialized, Role: FOLLOWER
I20260927 07:03:57.267135 26063 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } }
I20260927 07:03:57.267217 26143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.247001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.3.57.60:42325)
0927 07:03:57.247037 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.247038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.251731 (+ 4693us) server_negotiation.cc:407] Connection header received
0927 07:03:57.251775 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.251776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.251791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.251857 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.260206 (+ 8349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.260897 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.262023 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.262321 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.262373 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.264117 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.264118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.264118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.264125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.265811 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.266194 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.266195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.266195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.266250 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.266442 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.266443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.266444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.266584 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.266698 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.267061 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.267131 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:57.267438 26063 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:03:57.267598 26037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47a8420e9666435ebfcba0b91164d8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b2934beb1b5e45eaa7a42b449dd44783"
I20260927 07:03:57.267706 26037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:03:57.268966 26037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a8420e9666435ebfcba0b91164d8a7 in term 1.
I20260927 07:03:57.270239 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:57.247324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.3.57.61:40681)
0927 07:03:57.247365 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.247367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.260605 (+ 13238us) server_negotiation.cc:407] Connection header received
0927 07:03:57.260696 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.260697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.260709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.260738 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.261410 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.262153 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.263458 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:57.263924 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:57.263978 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:57.265879 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:03:57.265880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:03:57.265881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:03:57.265888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:03:57.267539 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.267961 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.267962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.267963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.268016 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:03:57.268612 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:03:57.268613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:03:57.268613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:03:57.269716 (+ 1103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:03:57.269806 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:57.270029 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 07:03:57.270128 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:57.271993 26196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:03:57.272049 26196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:03:57.273653 26112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [term 1 FOLLOWER]: Refusing update from remote peer 47a8420e9666435ebfcba0b91164d8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:57.273695 26037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [term 1 FOLLOWER]: Refusing update from remote peer 47a8420e9666435ebfcba0b91164d8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:03:57.274045 26146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:57.274176 26179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:03:57.274453 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.274608 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.275828 26160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a8420e9666435ebfcba0b91164d8a7. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.275916 26160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.276037 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a8420e9666435ebfcba0b91164d8a7. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.276108 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.279063 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.279178 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:57.279275 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.279342 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:03:57.279420 26183 mvcc.cc:205] Tried to move back new op lower bound from 7333857842265096192 to 7333857842167001088. Current Snapshot: MvccSnapshot[applied={T|T < 7333857842265096192}]
I20260927 07:03:57.279716 26198 mvcc.cc:205] Tried to move back new op lower bound from 7333857842265096192 to 7333857842167001088. Current Snapshot: MvccSnapshot[applied={T|T < 7333857842265096192}]
I20260927 07:03:57.279803 26160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.279872 26160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.279847 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47a8420e9666435ebfcba0b91164d8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a8420e9666435ebfcba0b91164d8a7" member_type: VOTER last_known_addr { host: "127.3.57.62" port: 40107 } } peers { permanent_uuid: "b2934beb1b5e45eaa7a42b449dd44783" member_type: VOTER last_known_addr { host: "127.3.57.61" port: 40681 } } peers { permanent_uuid: "7012bc2239f445fa8b7c8686b8a0d777" member_type: VOTER last_known_addr { host: "127.3.57.60" port: 42325 } } }
I20260927 07:03:57.279920 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:03:57.280000 26200 mvcc.cc:205] Tried to move back new op lower bound from 7333857842265096192 to 7333857842167001088. Current Snapshot: MvccSnapshot[applied={T|T < 7333857842265096192}]
I20260927 07:03:57.282718 26182 catalog_manager.cc:1347] Generated new cluster ID: a8a80779ea564eeb946c79069376f2cc
I20260927 07:03:57.282778 26182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:03:57.289628 26182 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:03:57.290423 26182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:03:57.307708 26182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47a8420e9666435ebfcba0b91164d8a7: Generated new TSK 0
I20260927 07:03:57.307932 26182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:03:57.337913 25991 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.336330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.3.57.62:40107)
0927 07:03:57.336379 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.336381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.337220 (+ 839us) server_negotiation.cc:407] Connection header received
0927 07:03:57.337346 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.337347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.337359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.337394 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.337840 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51468: BlockingRecv error: recv got EOF from 127.0.0.1:51468 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 07:03:57.338716 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.336759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.3.57.61:40681)
0927 07:03:57.336803 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.336805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.338156 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 07:03:57.338258 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.338259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.338271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.338303 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.338654 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50994: BlockingRecv error: recv got EOF from 127.0.0.1:50994 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 07:03:57.339570 26143 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.337098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.3.57.60:42325)
0927 07:03:57.337139 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.337141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.338966 (+ 1825us) server_negotiation.cc:407] Connection header received
0927 07:03:57.339089 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.339090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.339101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.339132 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.339490 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40766: BlockingRecv error: recv got EOF from 127.0.0.1:40766 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:03:57.341929 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.341144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.3.57.61:40681)
0927 07:03:57.341180 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.341182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.341406 (+ 224us) server_negotiation.cc:407] Connection header received
0927 07:03:57.341516 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.341517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.341535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.341566 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.341878 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50998: BlockingRecv error: recv got EOF from 127.0.0.1:50998 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:57.345054 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.344224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.3.57.61:40681)
0927 07:03:57.344258 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.344261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.344525 (+ 264us) server_negotiation.cc:407] Connection header received
0927 07:03:57.344644 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.344645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.344656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.344688 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.345005 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51014: BlockingRecv error: recv got EOF from 127.0.0.1:51014 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 07:03:57.349041 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.348198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.3.57.61:40681)
0927 07:03:57.348233 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.348235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.348491 (+ 256us) server_negotiation.cc:407] Connection header received
0927 07:03:57.348592 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.348593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.348604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.348635 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.348990 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51020: BlockingRecv error: recv got EOF from 127.0.0.1:51020 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 07:03:57.355077 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.354211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.3.57.61:40681)
0927 07:03:57.354251 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.354253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.354521 (+ 268us) server_negotiation.cc:407] Connection header received
0927 07:03:57.354626 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.354627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.354638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.354670 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.355022 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51036: BlockingRecv error: recv got EOF from 127.0.0.1:51036 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:03:57.365357 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.364428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.3.57.61:40681)
0927 07:03:57.364466 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.364468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.364752 (+ 284us) server_negotiation.cc:407] Connection header received
0927 07:03:57.364860 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.364861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.364872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.364905 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.365285 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51040: BlockingRecv error: recv got EOF from 127.0.0.1:51040 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 07:03:57.383827 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.382766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.3.57.61:40681)
0927 07:03:57.382813 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.382815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.383125 (+ 310us) server_negotiation.cc:407] Connection header received
0927 07:03:57.383246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.383247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.383260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.383294 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.383763 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 07:03:57.418135 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.417181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.3.57.61:40681)
0927 07:03:57.417225 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.417227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.417528 (+ 301us) server_negotiation.cc:407] Connection header received
0927 07:03:57.417635 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.417636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.417649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.417679 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.418074 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51048: BlockingRecv error: recv got EOF from 127.0.0.1:51048 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 07:03:57.484308 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.483660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.3.57.61:40681)
0927 07:03:57.483736 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.483739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.483813 (+ 74us) server_negotiation.cc:407] Connection header received
0927 07:03:57.483846 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.483847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.483857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.483923 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.484228 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51054: BlockingRecv error: recv got EOF from 127.0.0.1:51054 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 07:03:57.614526 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.613844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.3.57.61:40681)
0927 07:03:57.613896 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.613902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.613946 (+ 44us) server_negotiation.cc:407] Connection header received
0927 07:03:57.614008 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.614009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.614019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.614068 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.614436 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51064: BlockingRecv error: recv got EOF from 127.0.0.1:51064 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 07:03:57.872289 26144 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:03:57.871731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.3.57.61:40681)
0927 07:03:57.871809 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:57.871810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:57.871822 (+ 12us) server_negotiation.cc:407] Connection header received
0927 07:03:57.871876 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:57.871877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:57.871888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:57.871952 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:03:57.872248 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51078: BlockingRecv error: recv got EOF from 127.0.0.1:51078 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 07:03:58.244695 26171 catalog_manager.cc:1259] Loaded cluster ID: a8a80779ea564eeb946c79069376f2cc
I20260927 07:03:58.244773 26171 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: loading cluster ID for follower catalog manager: success
I20260927 07:03:58.245594 26171 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: acquiring CA information for follower catalog manager: success
I20260927 07:03:58.245921 26171 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7012bc2239f445fa8b7c8686b8a0d777: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:03:58.272464 26196 catalog_manager.cc:1259] Loaded cluster ID: a8a80779ea564eeb946c79069376f2cc
I20260927 07:03:58.272548 26196 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: loading cluster ID for follower catalog manager: success
I20260927 07:03:58.273307 26196 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: acquiring CA information for follower catalog manager: success
I20260927 07:03:58.273550 26196 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b2934beb1b5e45eaa7a42b449dd44783: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:03:58.388399 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:58.385611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.3.57.61:40681)
0927 07:03:58.385734 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:58.385736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:58.385785 (+ 49us) server_negotiation.cc:407] Connection header received
0927 07:03:58.385863 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:58.385864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:58.385942 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:58.386039 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:58.386191 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.386681 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.387501 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.387748 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.387840 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:58.388024 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:58.388132 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:58.388233 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:58.388273 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 07:03:58.388295 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260927 07:03:58.392380 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:58.390172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.3.57.60:42325)
0927 07:03:58.390276 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:58.390277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:58.390328 (+ 51us) server_negotiation.cc:407] Connection header received
0927 07:03:58.390403 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:58.390404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:58.390437 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:58.390499 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:58.390569 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.391038 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.391773 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.391955 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.392006 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:58.392130 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:58.392212 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:58.392259 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:58.392310 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 07:03:58.392324 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260927 07:03:58.396911 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:58.394014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.3.57.61:40681)
0927 07:03:58.394050 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:58.394052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:58.394205 (+ 153us) server_negotiation.cc:407] Connection header received
0927 07:03:58.394292 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:58.394293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:58.394302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:58.394353 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:58.394475 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.395024 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396065 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.396366 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396431 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:58.396616 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:58.396648 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:58.396696 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:58.396792 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 07:03:58.396817 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 07:03:58.397192 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:58.394118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.3.57.60:42325)
0927 07:03:58.394153 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:58.394155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:58.394164 (+ 9us) server_negotiation.cc:407] Connection header received
0927 07:03:58.394214 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:58.394215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:58.394225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:58.394278 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:58.394555 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.395249 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396210 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.396488 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396557 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:58.396699 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:58.396727 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:58.396776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:58.397106 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 07:03:58.397131 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 07:03:58.397418 26282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:03:58.393906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.3.57.62:40107)
0927 07:03:58.394023 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 07:03:58.394065 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0927 07:03:58.394213 (+ 148us) server_negotiation.cc:407] Connection header received
0927 07:03:58.394256 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:03:58.394257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:03:58.394306 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:03:58.394376 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:03:58.394517 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.395288 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396341 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:03:58.396664 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:03:58.396721 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:03:58.396923 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:03:58.397030 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:03:58.397084 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 07:03:58.397236 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:03:58.397306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260927 07:03:58.399233 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25920
I20260927 07:03:58.404692 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 25995
I20260927 07:03:58.409667 3300 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskaZGoNp/build/release/bin/kudu with pid 26070
2026-09-27T07:03:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2814 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5649 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (296443 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS